PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs
Authors: James Oldfield
Christos Tzelepis
Yannis Panagakis
Mihalis Nicolaou
Ioannis Patras
Venue: ICLR 2023 poster (2023)
Keywords: GANs
interpretability
local image editing

Abstract: Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing on the rich semantics that are embedded in the latent spaces of pre-trained GANs. However, existing methods are often tailored to specific GAN architectures and are limited to either discovering global semantic directions that do not facilitate localized control, or require some form of supervision through manually provided regions or segmentation masks. In this light, we present an architecture-agnostic approach that jointly discovers factors representing spatial parts and their appearances in an entirely unsupervised fashion. These factors are obtained by applying a semi-nonnegative tensor factorization on the feature maps, which in turn enables context-aware local image editing with pixel-level control. In addition, we show that the discovered appearance factors correspond to saliency maps that localize concepts of interest, without using any labels. Experiments on a wide range of GAN architectures and datasets show that, in comparison to the state of the art, our method is far more efficient in terms of training time and, most importantly, provides much more accurate localized control. Our code is available at: https://github.com/james-oldfield/PandA.

Published as a conference paper at ICLR 2023
PANDA: UNSUPERVISED LEARNING OF PARTS AND APPEARANCES IN THE FEATURE MAPS OF GANS
James Oldfield Christus Tzelepis', Yannis Panagakis
Mihalis A. Nicolaou", Ioannis Patras'
Queen Mary University of London, University of Athens, Thhe Cyprus Institute
ABSTRACT
Recent advances in the understanding of Generative Adversarial Networks (GANs) have led to remarkable progress in visual editing and synthesis tasks, capitalizing the rich semantics that are embedded in the latent spaces of pre-trained GANs. However, existing methods are often tailored to specific GAN architectures and are limited either discovering global semantic directions that do not facilitate localized control, or require some form of supervision through manually provided regions or segmentation masks. this light, we present an architecture-agnostie approach that jointly discovers factors representing spatial parts and their appear- ances in an entirely unsupervised fashion. These factors are obtained by applying semi-nonnegative tensor factorization the feature maps, which in turn enables context-aware local image editing with pixel-level control. In addition, we show that the discovered appearance factors correspond saliency maps that local- ize concepts interest, without using any labels. Experiments on a wide range of GAN architectures and datasets show that, in comparison the state of the art, our method is far more efficient in terms of training time and, most impor- tantly, provides much more accurate localized control. Our code available https:/ /ttpss:/ggiee ///g/giitige httpbbb.tttttg/gg//g com/ james o lf weld/P/a PanddA.
INTRODUCTION
Unsupervised learning parts appearances
Original
StyleGAN
StylEGAN
Original
Apparanas
BigGAN
BigGAN
Learnt
Part
(3) Local image editing
(b) Cantext Cantext-aware aware object removal
(c) Pixel-level concept localization
Figure 1: We propose unsupervised method for learning set of factors that correspond to interpretable parts and appearances dataset of images. These can used multiple tasks: (a) local image editing, (b) context-aware object removal, and (c) producing saliency maps for learnt concepts of interest.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) constitute the state of the art (SOTA) for the task of image synthesis. However, despite the remarkable progress in this domain through improvements to the image generator's 's architecture (Radford et al., 2016; Karras et al., 2018; 2019; 2020b; 2021; Brock et al., 2019), their inner workings remain to large extent unexplored. Developing better understanding of the way in which high-level concepts represented and composed to form synthetic images is important for a number of downstream tasks such as generative
"Corresponding author: j.a.o"d oldff d@ ac.uk

---

Published as a conference paper at ICLR 2023
model interpretability (Shen et al., 2020a; Bau et al., 2019; Yang et al., 2021) and image editing (Härkönen et al., 2020; Shen & Zhou, 2021; Shen et al., 2020c; Voynov & Babenko, 2020; Tzelepis et al., 2021; Bau et al., 2020). In modern generators however, the synthetic images are produced through an increasingly complex interaction of a set of per-layer latent codes in tandem with the feature maps themselves (Karras et al., 2020b; 2019; 2021) and/or with skip connections (Brock et al., 2019). Furthermore, given the rapid pace at which new architectures are being developed, demystifying the process by which these vastly different networks model the constituent parts of an image is an ever wer--peeeen challenge. Thus, many recent advances are architecture-speet specific (Wu et al., 2021; Collins et al., 2020; Ling et al., 2021) and a manipulating convolutional generators remains elusive.
method for analyzing and
popular line of GAN-based image editing research concerns itself with learning so-called "inter- pretable directions" in the generator's s latent space (Härkönen et al., 2020; Shen & Zhou, 2021; Shen at, 2020c; Voynov & Babenko, 2020; Tzelepis et al., 2021; Yang et al., 2021; He et al., 2021; Haas et al, 2021; 2022). Once discovered, such representations of high-level concepts can be manipulated to bring about predictable changes to the images. One important question in this line of research is how latent representations
combined to form the appearance a particular local region of the im- age. Whilst some recent methods attempt to tackle this problem (Wang et al., 2021; Wu et al., 2021; Broad et al., 2022; Zhu et al., 2021a; a;; Zhang et al., 2021; Ling et al., 2021; Kafri et al., 2021), the cur- rent state-of-the-art methods come with a number of important drawbacks and limitations. In particu- lar, existing techniques require prohibitively long training times (Wu al., 2021; Zhu al., 2021a), costly Jacobian- based optimization (Zhu et al., 2021a; a;; 2022), and the requirement of semantic masks (Wu et al., 2021) or manually specified regions of interest (Zhu al., 2021 a; 2022). Furthermore, whilst these methods successfully find directions affecting local changes, optimization must be per- formed on a per-region basis, and the resulting directions do not provide pixel-level control-a a term introduced by Zhu et al. (2021 a) referring to the ability to precisely target specific pixels in the image.
In this light, we present a fast unsupervised method for jointly learning factors for interpretable parts and their appearances (we thus refer to our method as PundA) in pre-trained convolutional generators. Our method allows one to both interpret and edit an image's s style at discovered local semantic regions of interest, using the learnt appearance representations. We achieve this by formulating constrained optimization problem with a semi- nonneiioneeettve tensor decomposition of the dataset of deep feature maps 2 E R M ff x W C in convolutional generator. This allows one to accomplish number of useful tasks, prominent examples of which are shown in Fig. 1. Firstly, our learnt representations of appearance across samples can be used for the popular task of local image editing (Zhu et al., 2021a; Wu et al., 2021) (for example, to change the colour or texture of a cat's ears as shown in Fig. 1 (a)). Whilst the satte-of-the-as art methods (Zhu et al., 2021a; Wu et al., 2021; Zhu et al., 2022) provide fine- grained control over a target region, they adopt an "annotation- firstt" approach, requiring an end-user to first manually specify a ROI. By contrast, our method fully exploits the unsupervised learning paradigm, wherein such concepts are discovered automatically and without any manual annotation. These discovered semantic regions can then be chosen, combined, or even modified by an end-user as desired for local image editing.
More interestingly still, through a generic decomposition of the feature maps our method identifies representations of common concepts (such as "background") in all generator architectures considered (all 3 StyleGANs (Karras et al., 2019; 2020b; 2021), ProgressiveGAN (Karras et al., 2018), and BigGAN (Brock et al., 2019)). This is surprising finding, given that these generators radically different in architecture. By then editing the feature maps using these appearance factors, we can thus, for example, remove specific objects in the foreground (Fig. (b)) in all generators, seamlessly replacing the pixels at the target region with the background appropriate to each image.
However, our method is useful not only for local image editing, but also provides a straightforward way to localize the learnt appearance concepts in the images. By expressing activations in terms of our learnt appearance basis, we are provided with a visualization of how much of each of the appearance concepts are present at each spatial location (i.c., saliency maps for concepts of interest). By then thresholding the values in these saliency maps (as shown in Fig. I (c)). we can localize the learnt appearance concepts (such as sky, floor, or background) in the images without the need for supervision at any stage.
We show exhaustive experiments on 5 different architectures (Karras et al., 2020b; 2018; 2021; 2019; Brock et al., 2019) and datasets (Deng et al., 2009; Choi et al., 2020; Karras et al., 2019; Yu et al.,
2

---

Published as a conference paper at ICLR 2023
Table 1: A high-level comparison of our method the SOTA for local image editing, "Training time"
denotes the total training time required to produce the images for the quantitative comparisons.
Manual ROI free
Semantic mask ftce
Pisel-level control
Architecture agnostic
Style diversity
Training time (mins)
StyleSpace (Was 2021) Low RankGAN (Zhu et ol,, 2021a)
177.12
X
8
X
524.21
ReSeFa al., Ours
347.79
/
J
0.87
2015; Karras et al., 2020a). Our method is not only orders of magnitude faster than the SOTA, but also showcases superior performance at the task of local image editing, both qualitatively and quantitatively. Our contributions can be summarized as follows:
We present architecture parts and the appearances of images in pre-trained GANs, that enables local image editing. In contrast to the SOTA (Zhu et al., 2021a; Wu et al., 2021), our approach requires neither semantic masks nor manually specified ROIs, yet offers more precise pixel-level control.
unsupervised framework for learning factors for both the
Through semi-nonnegative tensor decomposition of the generator's ' feature maps, we show how one can learn sparse representations of semantic parts of images by formulating and solving an appropriate constrained optimization problem.
We show that the proposed method learns appearance factors that correspond to semantic concepts (e.g., background, sky, skin), which can localized in the image through saliency maps.
A rigorous set of experiments show that the proposed approach allows for more accurate local image editing than the SOTA, while taking only a fraction of the time to train.
2 RELATED WORK
Generative Adversarial Networks (GANs) (Goodfellow al., 2014) continue to push forward the state of the art for the task of image synthesis through architectural advances such as the use of convolutions (Radford al., 2016), progressive growing (Karras al., 2018), and style-based architectures (Karras et al., 2019; 2020b; 2021). Understanding the representations induced by these networks for interpretation (Bau et al., 2019; Shen et al., 2020a; Yang et al., 2021) and control (Shen & Zhou, 2021; Härkönen et al., 2020; Voynov & Babenko, 2020; Georgopoulos et al., 2021; Tzelepis et al., 2021; Zhu et al., 2021a; Bounareli et al., 2022; Wu et al., 2021; Abdal et al., 2021) has subsequently received much attention.
However, whilst several methods identify ways of manipulating the latent space of GANs to bring about global semantic changes- in supervised (Goetschalckx et al., 2019; Plumerault et al., 2020; Shen et al., 2020c;a) or unsupervised (Voynov Babenko, 2020; Shen & Zhou, 2021; Härkönen et al., 2020; Tzelepis et al., 2021; Oldfield et al., 2021) manner-many of them struggle to apply local changes to regions of interest in the image. In this framework of local image editing, one can swap certain parts between images (Collins et al., 2020; Jakoel et al., 2022; Chong al., 2021; Suzuki et al., 2018; Kim et al., 2021; Bau et al., 2020), or modify the style at particular regions (Wang et al., 2021; Wu et al., 2021; Broad et al., 2022; Zhu et al., 2021a; Zhang et al., 2021; Ling et al., 2021; Kafri et al., 2021). This is achieved with techniques such as clustering (Collins et al., 2020; Zhang al., 2021; Broad et al., 2022; Kafri et al., 2021), manipulating the AdaIN (Huang & Belongie, 2017) parameters (Wu et al., 2021; Wang et al., 2021), or/and operating on the feature maps themselves (Wang al., 2021; Broad et al., 2022: Zhang et al., 2021) to aid the locality of the edit. Other approaches employ additional latent spaces or architectures (Kim et al., 2021; Ling et al., 2021), require the computation of expensive gradient maps (Wang et al., 2021; Wu al., 2021) and semantic segmentation masks/networks (Wu et al., 2021; Zhu et al., 20216; Ling et al., 2021). or require manually specified regions of interest (Zhu et al., 202 2022). In contrast to related work, our method automatically learns both the parts and a diverse set of global appearances, in fast unsupervised procedure without any semantic masks. Additionally, our method allows for pixel-level control (Zhu et al., 2021a). For example, one choose to modify a single eye only in face, which is not possible with the SOTA (Zhu et al., 202 la; 2022). Our method and its relationship to the SOTA for local image editing is summarized in Table 1.
3

---

Published as a conference paper at ICLR 2023
Appearance
a
a
a,
factors
Multiplicative interactions
Rc
Zi
Parts
W
Air1
Xir2 P2
XirRs PRs factors
Figure 2: An overview of our method. We decompose dataset of generator's activations Z, R R XC XC with separable model. Each factor has an intuitive interpretation: the factors for the spatial modes Pj control the parts, determining which spatial locations in the feature maps the various appearances ag are present, through their multiplicative interactions.
From a methodological standpoint, most closely related to our method are the works of Collins et al. (2020; 2018). Both of these perform clustering in the activation space for parts-based representations in generators (Collins et al., 2020) and CNNs (Collins et al., 2018) respectively. However, Collins al. (2018) considers only discriminative networks for locating common semantic regions in CNNs, whilst we additionally focus on image editing tasks in GANs. On the other hand, Collins et al. (2020) does not jointly learn representations of uppearances. Therefore Collins et al. (2020) is limited to swapping parts between two images, and is additionally StyleGAN-specific, unlike our method that offers generic treatment convolutional generators.
METHODOLOGY
In this section, detail our approach for jointly learning interpretable parts and their appearances in pre-trained GANs, in an unsupervised manner. We begin by establishing the notation used throughout the paper in Section 3.1. We then introduce our proposed separable model in Section 3.2, and our optimization objective in Section 3.3. In Section 3.4 we describe our initialization strategies, and fimally in Section 3.5 we describe how refine the learnt global parts factors.
3.1 NOTATION
We use uppercase (lowercase) boldface letters to refer to matrices (vectors), e.g., X (x), and calli- graphic letters for higher-order tensors, e.g., X. We refer each element an N order tensor X' E x/xx/xxxxxxx using N indices, i.e., (i, ........N) A iz.i R. The mode-n fibers of tensor are the column vectors formed when fixing all but the n mode's indices (e.g., X.jk E R4: are the mode-1 fibers). A tensor's mode-n fibers can be stacked along the columns a matrix, giving us the mode-n unfolding denoted as X(x) E Iax, with I, - I, (Kolda & Bader, 2009).
tgn
We denote a pre-trained convolutional GAN generator with G. and use G(ej refer to the partial application of the last layers of the generator only.
3.2 SEPARABLE MODEL. OF PARTS AND APPEARANCES
convolutional generator maps each latent code z, .V (O. synthetic image X, R fxixx via sequence of 2D transpose convolutions. The intermediate convolutional features 2, E R HxXC each layer have a very particular relationship to the output image. Concretely, each spatial activation (Olah et al., 2018) (which can be thought of spatial coordinate in the feature maps in Fig. 2 indexed with an (h, w) tuple) affects a specific patch in the output image (Collins et al., 2020). At each of these spatial positions, channel fiber Zıhu: BCC lies depth-wise along the activation tensor, determining its value. With this understanding, we propose to factor the spatial and channel modes separately with tensor decomposition, providing intuitive separation into representations of the images' parts and appearances. This provides a simple interface for local image editing. We suggest that representations of set of interpretable parts for local image editing should have two properties:
Non-negatinn nen-atiitiiiyy: the representations ought to be additive in nature, thus corresponding to semantic parts of the images (Lee & Seung, 1999).
4

---

Published as a conference paper at ICLR 2023
2. Sparsity: the parts should span disjoint spatial regions, capturing different localized patterns in space, as opposed to global ones (Yang & Oja, 2010; Yuan et al., 2009).
Concretely, given the dataset's intermediate feature maps 2 E R-N*HxWx from the pre-trained generator, each sample's 's mode-3 unfolding Zu(3) E CR contains in columns the channel-wise activations at each of the S H W spatial positions in the feature maps." We propose separable factorization of the form
Z((3) AA,P
AELi
A112
Pi
a1
R
(2)
AiRe1
Rs Rs
PRa
Appearance
Sample i's coefficients
Parts
where A E RCxRe are the global appearance factors and P 2 0 RSxRR are the global parts factors (with Rc I C, Rs I S), jointly learnt across many samples dataset. Intuitively, the coefficients Äijk encode how much of appearance aj is present at part Ps: in sample i's feature maps (3). We show our proposed separable decomposition schematically in Fig. Each non- negative parts factor P& E R 2 O spans a spatial sub-region of the feature maps, corresponding to a semantic part. The various appearances and textures present throughout the dataset the appearance factors aj E C and lie along the depth-wise channel mode of the feature maps. This formulation facilitates modelling the multiplicative interactions (Jayakumar al., 2020) between the parts and appearance factors. Concretely, due to the outer product, the factors relating to the parts control the spatial regions which the various appearance factors are present. The parts factors thus function similarly to semantic masks, but rather are learnt jointly and in an entirely unsupervised manner. This is particularly useful for datasets for which segmentation masks are not readily available.
encoded in
3.3 OBJECTIVE
We propose to solve constrained optimization problem that leads to the two desirable properties outlined in Section 3.2. We impose hard non-negativity constraints on the parts factors P to achieve property 1, and encourage both factor matrices to be column-orthonone has been shown to lead to sparser representations (Ding et al., 2006; Yang & Laaksonen, 2007; Yang & Oja, 2010; Yuan et al., 2009), and has intricate connections clustering (Ding al., 2005; Li & Ding, 2006)). We achieve this by formulating a single reconstruction objective as follows. Let 2 NxC' be batch of N samples' mode-3 unfolded intermediate activations. Then constrained optimization problem is
for property (which
min ((Z,A.P) = min Σ - A.P
S.t. S.t. O.
1=1
good reconstruction naturally leads to orthogonal factor matrices (e.g., P R In for PE gS with S > Rs) without the need for additional hard constraints (Le et al., 2013). What's more, each parts factor (column of P) encouraged to span a distinct spatial region to simultancously satisfy both the non-negativ and problem is non-convex. We thus propose to break the problem into two sub-problems A and P separately, applying a form of block-coordinal descent (Lin, 2007), optimizing each factor matrix separately whilst keeping the other fixed. The gradients of the objective function in Eq. (3) with respect the two factor matrices (see the supplementary material for the derivation) are given by
constraints. However, this
VpL 2
Σ
PZ. AÄZP + Z AAZ, 2Z AZ,
(4)
VAC= 2
Σ
A AA 22.P2. A
(5)
'Intuitively, Zuz) RCx8 can be viewed simply 'reshaping' of the th sample's feature maps that combines the height and width modes into single S-dimensional 'spatial' mode.
5

---

Published as a conference paper at ICLR 2023
with P PP and A a AAT. After gradient update for the parts factors P, project them
onto the non -engetttvve orthant (Lin, 2007) with max {0, This leads to our alternating optimization algorithm, outlined in Algorithm
Algorithm 1: Block--cordinne descent solution to Eq. (3)
Input :2 E RMMxXxS (M lots of mode-3-unfolded activations), Rc, Rs E IR (ranks), (learning rate), and T (# iterations).
Output:P RRRRR, E CxRR (parts and appearance factors).
Initialise
U.E, V + SVD (1))
( - U.
P(I) -0(00,000))
for f 1 do
}
p(++) max
0. ((t) - X. Ppn (2, A (2))),
// PGD step
A (++1) A(0) V p(t-1));
end
Upon convergence of Algorithm 1, to modify an image at region k with the j appearance with desired magnitude a IR, we compute the forward pass from layer onwards in the generator with X Gpa (Z. + aa, with Py: being the normalized parts factor of interest.
3.4 INITIALIZATION
Let Z NNxX a batch of mode-3 unfolded feature maps as Section 3.3. A common initialization strategy (Cichocki et al., 2009; Boutsidis & Gallopoulos, 2008; Yuan et al., 2009) for non-negative matrix/tensor decompositions via a form of HOSVD (Tucker, 1966; Lu et al., 2008). Without non-negativity constraints, the channel factor matrix subproblem has closely related closed- form solution given by the first Re left-singular vectors of the mode-2 unfolding the activations expressed in terms of the parts basis (proof given in Appendix of Xu et al. (2005)). We thus initialize the channel factors at time-step t 1 with A (1) & U. where U.R. the first Re- many left-singular vectors of Z(2) 4/2y Later on in Section 4. 4.2 we demonstrate the benefits of this choice, including its usefulness for locating interpretable appearances.
3.5 PARTS FACTORS REFINEMENT
The formulation in Eq. (3) for learning parts and appearances makes the implicit assumption that the samples are spatially aligned. However, this does not always hold in practice, and therefore the global parts are not always immediately useful for datasets with no aligmment. To alleviate this requirement, we propose fast optional "refinement" step of the global parts factors P E Sx from Eq. (3) to specialize them to sample-specific parts factors P, E R RS R. for sample i. Given the tth target sample's intermediate activations Z; E CCS we optimize a few steps of a similar constrained optimization problem as before:
min L Z, A, P,) = min Z, A Z.P. P. s.t. P, 2 0. P.
(6)
We analyze in Appendix B.0.2 the benefits of this refinement step, and compare the global parts factors to the refined factors.
EXPERIMENTS
In this section present series of experiments validate the method and explore its properties. We begin in Section 4.1 by focusing on using the method for interpretation: showing how one generate saliency maps for concepts of interest and remove the foreground at target locations.

---

Published as a conference paper at ICLR 2023
Following this, showcase local image editing results 5 GANs in Section 4.2, In Appendix B we present ablation studies to further justify and motivate our method.
4.1
INTERPRETING THE APPEARANCE VECTORS
StyleGAN1
StyleGAN2
StyleGAN3
BigGAN
PGGAN
Original
Edited
Part
Figure 3: Our architecture architecture-agnostie method discovers representation of the "background" concept the feature maps, which allows us to remove objects in a context-aware manner for all 5 generators.
Using the learnt appearance basis A, one can straightforwardly visualize "how much" of each column present each spatial location via a change of basis. In particular, the element at row and column 8 of the activations expressed in terms of the appearance basis A Z; E. IR Rc xS encodes how much of appearance is present at spatial location s. for particular sample of interest. This transformation provides a visual understanding of the concepts controlled by the columns by observing the semantic regions in the image which these values are the highest.
4.1.1 GENERIC CONCEPTS SHARED ACROSS GAN ARCHITECTURES
The analysis above leads make interesting discovery. We find that model frequently learns an appearance vector for a high-level "background" concept in all 5 generator architectures. This is surprising finding-one would not necessarily expect these radically different architectures to encode concepts in the same manner (given that many existing methods are architecture-specfffc) let alone that they could be extracted with a single unsupervised approach. We can thus use this learnt "background" appearance vector to remove objects in a context-aware manner, as shown on all generators and numerous datasets in Fig. 3.
4.1.2 VISUALIZING AND LOCALIZING APPEARANCE VECTORS
layer: appeerance:
layer: appearance:
layer: appearance:
layer: 6, appearance:
Original
Projection
Mask
(a) StyleGAN2 (AFHQ dog)
(b) StyleGAN2 (FFHQ)
(c) BigGAN (breskwater)
(d) BigGAN (promontory)
Figure 4: Visualizing the coordinates in the appearance basis (2nd row), one can interpret how much of each appearance vector present at each spatial patch. For example, we see appearance vectors at various layers very clearly corresponding to (a) background, (b) skin, (c) sky, and (d) foreground.
Through the change of basis A Z; the concept & of interest (e.g., the "background" concept), offering an interpretation of the images' semantic content. We first compute the saliency map m, a. Z, R$. whose elements encode the magnitude of concept & at each spatial position in the 5th sample. This can be reshaped into
identify the pixels in the image that composed of
7

---

Published as a conference paper at ICLR 2023
square matrix and visualized an image to localize the hthh concept in the image, shown in row of Fig. 4. We then additionally perform a simple binary classification following Voynov & Babenko (2020). We classify each pixel an instance of concept & not with maj [makj Mk], where 14k N.S En.s Minks R is the mean magnitude of the Tthh concept in N samples. We show this in row 3 of Fig. 4 for various datasets and GANs. For example, this analysis allows to identify (and localize) appearance vectors in various generators that control concepts including "foreground", "skin", and "sky". shown in Fig. 4 (b-d). We find this visualization to be most useful for understanding the first few columns of A, which control the more prominent high-level visual concepts in the dataset due to SVD-based initialization outlined in Section 3.4.
Original Part(s)
Modified appearance
Original
Part(s)
Modified appearance
BigGAN
StyleGAN2
Local image editing on a number of architectures and datasets, using both the global and refined parts factors. At each column, the original image is edited at the target part with different appearance vector (many more examples are shown in Appendix C.3).
Figure 5:
4.2 LOCAL IMAGE EDITING
Next, we showcase our method's ability to perform local image editing in pre-trained GANs, on generators and 5 datasets (ImageNet (Deng et al., 2009), AFHQ (Choi et al., 2020), FFHQ (Karras 2019), LSUN (Yu et cl., 2015), and MetFaces (Karras et al., 2020a)). In Fig. 5 we show number of interesting local edits achievable with method, using both the global and refined parts factors. Whilst we can manipulate the style at common regions such as the eyes with the global parts factors, the refined parts factors allow one target regions such as individual's clothes, their background. Many more examples are shown in Appendix C.3. One is not limited to this set of learnt parts however. For example, one can draw ROI by hand test-time modify an existing part-an example of this is shown in Appendix C.I. This way, pixel-level control (e.g., opening only a single eye of face) is achievable in a way that is not possible with the SOTA methods (Zhu al., 202 Wu et al., 2021).
We next compare our method to state-of-the-eee art GAN-based image editing techniques in Fig. 6. In particular, we train our model at layer using Rs 8 global parts factors, with refinement. As can be seen, SOTA methods such as LowRank-GAn (Zhu et al., 202 excel at enlarging the eyes photo-realistic manner. However, we frequently find the surrounding regions to change as well. This is seen clearly by visualizing the mean squared error (Collins et al., 2020) between the original images and their edited counterparts, shown in every second row of Fig. 6. We further quantify this ability to affect local edits in the section that follows.
4.2.1 QUANTITATIVE RESULTS
We compute the ratio of the distance between the pixels of the original and edited images in the region of 'disinterest', over the same quantity with the region of interest:
(1
M) * (2; - x;
ROIR(M, X.x' N
MM
(x-x
8

---

Published as a conference paper at ICLR 2023
Original
GANSpace
SeFa
StyleSpace
LowRankGAN
ReSeFa
Ours
Edits
MSE
Figure 6: Qualitative comparison to SOTA methods editing the "eyes" ROI. We also show the mean squared error (Collins et al., 2020) between the original images and their edited counterparts, highlighting the regions that change.
Table 2: ROIR (L) of Eq. (7) for 10k FFHQ samples per local edit.
Eyes
Nose
Open mouth
Smile
GANSpace (Härkönen et al., 2020)
2.80
4.89-2.11
1.33
2.44=0.89
SeFu (Shen & Zhou, 2021)
5.01
6.89+3.04
3.45+11112
5.04=2.22
StyleSpace (Wu et al,, 2021)
1.26+0.70
1.70=0.82
1.24++0.4
2.06=1.62
LowRankGAN (Zhu et al., 2021a)
1.7810.59
5.07-2.06
1.82:10.60
2.31-0.76
ReSeFa (Zhu et al., 2022)
2.2110.85
2.92+1.29
1.69100665
1.87=0.75
Ours
1.04:10.33
1.17
1.04:10.39
1.05-0.38
where M 1 is H W spatial mask (replicated along the channel mode) specifying the region interest, 1 is a 1-tensor, and X,X' Nx xC are the batch of original and edited versions of the images respectively. A small ROIR indicates more 'local' edits, through desirable change to the ROI (large denominator) and little change elsewhere (small numerator). We compute method and SOTA baselines in Table 2, for number of regions of interest. As can
this metric for be seen, our method consistently produces more local edits than the SOTA for a variety of regions of interest. We posit that the reason for this is due our operating directly the feature maps, where the spatial activations have a direct relationship to a patch in the output image. Many more comparisons and results can be found in Appendix C.
CONCLUSION
In this paper, we have presented a fast unsupervised algorithm for learning interpretable parts and their appearances in pre-trained GANs. We have shown experimentally how our method outperforms the state of the art at the task of local image editing, in addition to being orders of magnitude faster to train. We showed how one can identify and manipulate generic concepts in generator architectures. We also believe that our method's s ability to visualize the learnt appearance concepts through saliency maps could be useful tool for network interpretability.
Limitations Whilst we have demonstrated that our method can lead to more precise control, the approach is not without its limitations. Such strictly local editing means that after modifying a precise image region, any expected influence on the rest of the image is not automatically accounted for. As concrete example, one can remove trees from an image, but any shadow they may have cast elsewhere is not also removed automatically. Additionally, we find that methods editing the feature maps have greater tendency to introduce artifacts relative to methods working on the latent codes. This is one potential risk with the freedom of pixel-level control-adding appearance vectors at arbitrary spatial locations does not always lead to photorealistic edits. We hope to address this in future work.
Acknowledgments
This work supported by the EU H2020 A14Media No. 951911 project.

---

Published as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
Efforts have been made throughout to ensure the results are reproducible, and the method easy to implement. We provide full source code in the supplementary material folder. This contains jupyter notebooks to reproduce the concept localizations from Section 4.1.2, the qualitative results from Section 4.2, and full demo including the training and refinement objective of Section 3.5 for easy training and editing with new generators. We also provide pre-trained models that can be downloaded by following the link in the supplementary material's 's readme... .nd.
ETHICS STATEMENT
The development of any new method such as PandA facilitating the ability to edit images brings a certain level of risk. For example, using the model, bad actors may more easily edit images for malicious ends spread misinformation. Additionally, it's important highlight that through our use of pre-trained models we inherit any bias present in the generators or the datasets on which they were trained. Despite these concerns, we believe our method leads more interpretable and transparent image synthesis-for for example, through our concept localization one has a much richer understanding of which attributes appear in the generated images, and where.
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. StyleFlow: Attribute-conditioned exploration of BylleGAN-ggnerater images using conditional continuous normalizing flows. ACM Trans. Graph., 40(3), May 2021.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. In Int. Conf. Learn. Represent., 2019.
David Bau, Steven Liu, Tongzhou Wang. Jun-Yan Zhu, and Antonio Torralba. Rewriting deep generative model. In Eur. Conf. Comput. Vis., pp. 351-369. Springer, 2020.
Stella Bounareli, Vasileios Argyriou, and Georgios Tzimiropoulos. Finding directions in gan's latent space for neural face reenactment, 2022.
C. Boutsidis and E. Gallopoulos. SVD based initialization: A head start for nonnegative matrix factorization. Pattern Recognition, 4
2008.
Terence Broad, F. Frederic Leymarie, and Mick Grierson. Network bending: Expressive manipulation of generative models in multiple domains. Entropy, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Ini. Conf. Mach. Learn., 2019.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 8188-8197, 2020.
Min Jin Chong, Wen-Sheng Chu, Abhishek Kumar, and David Forsyth. Retrieve in style: Unsuper- vised facial feature transfer and retrieval. In Int. Conf. Comput. Vis., October 2021.
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun ichi Amari. Nonnegative Matrix and Tensor Factorizations Applications Exploratory Multi-way Data Analysis and Blind Source Separation. Wiley, 2009. ISBN 978-0-470-74727-8.
Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery. In Eur. Conf. Comput. Vis., pp. 336-352, 2018.
Edo Collins, Raja Bala, Bob Price, and Sabine Süsstrunk. Editing in style: Uncovering the local semantics of GANs. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Patter Recog, pp. 248-255 2009.
10

---

Published as a conference paper at ICLR 2023
Chris Ding, Xiaofeng He, and Horst D Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM international conference on data mining, pp. 606-610. SIAM, 2005.
Chris Ding, Tao Li, Wei Peng, and Haesun Park. Orthogonal nonnegative matrix t-factorizations for clustering. In ACM SICKDD Int. Conf. Knowledge Discovery and Data Mining, KDD '06, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395.
Markos Georgopoulos, James Oldfield, Grigorios Chrysos, and Yannis Panagakis. Cluster-guided image synthesis with unconditional models, 2021.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions of cognitive image properties. In Int. Conf. Comput. Vis., pp. 5744-5753, 2019.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Marde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Adv. Neural Inform. Process. Syst., 2014.
René Haas, Stella Graßhof, and Sami S. Brandt. Tensor-based subspace factorization for StyleGAN. In Proc. Int. Conf. Automatic Face and Gesture Recognit. (FG), 2021.
René Haas, Stella Graßhof, and Sami S. Brandt. Tensor-based emotion editing in the StyleGAN latent space. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh.. 2022.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering interpretable GAN controls. In Ady. Neural Inform. Process. Syst., volume 33, pp. 9841-9850, 2020.
Zhenliang He, Meina Kan, and Shiguang Shan. Eigengan: Layer-wise eigen-learning for gans. Int. Conf. Comput. Vis., pp. 14408-14417, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. Adv Newral Inform. Process. Syst., 30, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Int. Conf. Comput. Vis., Oct 2017.
Karin Jakoel, Liron Efraim, and Tamar Rott Shaham. GANs spatial control via inference-time adaptive normalization. In IEEE Winter Conf. Applic. Comput. Vis., January 2022.
Siddhant M. Jayakumar, Wojciech M. Czarnekki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In Int. Conf. Learn. Represent. 2020.
Omer Kafri, Or Patashnik, Yuval Alaluf, and Daniel Cohen-Or. StyleFusion: A generative model for disentangling spatial segments, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Int. Conf. Learn. Represent. 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., June 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Adv. Neural Inform. Process. Syst., 33:12104- 12114, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Puttern Recog. June 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Adv. Neural Inform. Process. Syst., 34, 2021.
11

---

Published as a conference paper at ICLR 2023
Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploiting spatial dimensions of latent in gan for real-time image editing. In JEEE Conf. Comput. Vis. Pattern Recog.. 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Tamara G. Kolda and Brett W. Bader. Tensor Decompositions and Applications. SIAM Review, 51 (3):455-500) August 2009.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. TensorLy: Tensor learning in Python. J. Mach. Learn. Research, 20(26), 2019.
Quoc V Le, Alexandre Karpenko, Jiquan Ngiam, and Andrew Y Ng. ICA with Reconstruction Cost for Efficient Overcomplete Feature Learning. In Adv. Neural Inform. Process. Syst. pp. 9, 2013.
Daniel D. Lee and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755))788-7911 October 1999. ISSN 0028-0836, 1476-4687.
Tao Li and Chris Ding. The relationships among various nonnegative matrix factorization methods for clustering. In Sixth International Conference on Data Mining (ICDM 06). pp. 362-371. IEEE, 2006.
Chih-Jen Lin. Projected Gradient Methods for Nonnegative Matrix Factorization. Neural Computa- tion, 19(10):756-2779, October 2007. ISSN 0899-7667, 1530-888X.
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. In Adv. Neural Inform. Process. Syst., 2021.
Haiping Lu, Konstantinos N Plataniotis, and Anastasios N Venetsanopoulos. Mpca: Multilinear principal component analysis of tensor objects. IEEE Trans. Neural Networks, 19(1):18-39, 2008
Luke Melas-Kyriazi, Christian Rupprecht, Laina, and Andrea Vedaldi. Finding an unsupervised image segmenter in each of your deep generative models. In Ini. Conf. Learn. Represent. 2022.
2022
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye. and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. http:/iillllpu///01/uuuiiiiiiii blocks.
James Oldfield, Markos Georgopoulos, Yannis Panagakis, Mihalis A. Nicolaou, and Ioannis Patras. Tensor Component Analysis for Interpreting the Latent Space of GANs. In Brit. Mach. Vis. Conf. November 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, hirh-performance deep learning library. In Adv. Neural Inform. Process. Syst., pp. 2019.
Curran Associates, Inc.,
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In Int. Conf. Learn. Represent. 2020.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition, 106:107404, 2020. ISSN 0031-32000 107404.
doi: https://doi.org/10. patco..atco
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Int. Conf. Learn. Represent. 2016.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in GANs. In IEEE Conf. Comput. Vis. Pattern Recog,, 2021.
12

---

Published as a conference paper at ICLR 2023
Yujum Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of GANs for semantic face editing. In IEEE Conf. Comput. Vis. Pattern Recog., 2020a.
Yujun Shen, Yinghao Xu, Ceyuan Yang, Jiapeng Zhu, and Bolei Zhou. Genforce. https:// :// 2020b.
github. cwm. gygeffrrceggeffoongo
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. InterFaceGAN: Interpreting the disentan- gled face representation learned by GANs. IEEE Trans. Pattern Anal. Mach. Intell., 2020c.
Ryohei Suzuki, Masanori Koyama, Takeru Miyato, Taizan Yonetsuji, and Huachun Zhu. Spatially controllable image synthesis with internal representation collaging, 2018.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3): 279-311, 1966.
Christos Tzelepis, Georgios Tzimiropoulos, and Ioannis Patras. WarpedGANSpaace Finding non- linear RBF paths in GAN latent space. In Int. Conf. Compur. Vis., pp.
October 2021.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In Int. Conf. Mach. Learn., pp. 9786-9796, 2020.
Andrey Voynov, Stanislav Morozov, and Artem Babenko. Big GANs are watching you: To- wards unsupervised object segmentation with off-the-shelf generative models. arXiv preprint arXiv:2006.04490 2020.
Rui Wang. Jian Chen, Gang Yu, Li Sun, Changqian Yu, Changxin Gao, and Nong Sang. Attribute- specific Control Units in StyleGAN for Fine-grained Image Manipulation. In ACM Int. Conf. Multimedia, October 2021.
Zongze Wu, Dani Lischinski, and Eli Shechtman. StyleSpace analysis: Disentangled controls for stylegan image generation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Dong Xu, Shuicheng Yan, Lei Zhang, Hong-Jiang Zhang, Zhengkai Liu, and Heung-Yeume Shum. Concurrent subspaces analysis. In IEEE Conf. Comput. Vis. Pallern Recog,, volume 2, June 2005.
Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges deep generative represen- tations for scene synthesis. Int. J. Comput. Vis., 129(5):1451-1466 2021.
Zhirong Yang and Jorma Laaksonen. Multiplicative updates for non-negative projections. Neurocom- puting, 71(1-3):363-77, 2007.
Zhirong Yang and Erkki Oja. Linear and nonlinear projective nonnegative matrix factorization. IEEE Trans. Neural Networks, 21(5):73--749, 2010.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop, 2015.
Zhijian Yuan, Zhirong Yang, and Erkki Oja. Projective nonnegative matrix factorization: Sparseness, orthogonality, and clustering. Neural Process. Lett, pp. 11-13, 2009.
Chen Zhang, Yinghao Xu, and Yujun Shen. Decorating your own bedroom: Locally controlling image generation with generative adversarial networks, 2021.
Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, and Qifeng Chen. Low-Rank Subspaces in GANs. In Adv. Neural Inform. Process. Syst., 2021a.
Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, and Qifeng Chen. Region-based semantic factorization in GANs. Int. Conf. Mach. Learn., 2022.
Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Barbershop: GAN-based Image Compositing using Segmentation Masks, ACM Trans. Graph., October 2021b.
13

---

Published as a conference paper at ICLR 2023
A GRADIENT'S
We first provide derivation of the gradients of the main paper's loss function with respect to each factor. The main paper's s objective function is given by
C
ZZ A(A AAATAAAAA)) ZPP)
N
tr (ZA:AA(AT,,)PP))) - (44-A(ATZ,PPP7))
i=1
N
tr
[Z Z:)
2tr
PP Z AA Z.)
+tr +tr(PP Z; AA AA
AA Z.P
f.
C.
Clearly, VpC. = 0. The term VpCo is of the form
oF [PP X). Thus have
VpL. (Z. AA Z AA Z.) P = 2 z AA Z,P.
The final term Pp has the form ap (PP XPPT) ). Through the chain rule, we have
VpL. 2 (PP (Z. AA AA Z, P (Z, AA PPTP)
(9)
Combining the terms, and defining & PPT and A & AA for convenience, the gradient of the reconstruction loss w/r/t the parts factors is thus
VpL 4 AA ZP) +2(PP AA AA Z AAAAAAA AA Z.PP AP
(10)
i=1
N Σ
2
PZ, AAZP 22 AAZPP - -2, AZ.
(11)
Via similar arguments, the gradient whv/t the channel factors is
N
VAC=
(Z,P z, A) + Z,P Z,P' 'PP PP z A ZPP'PP'Z'AA'A
(12)
i=1
N Σ
ÁZ,PPZ A ZPPZ, AA 2Z.PZ, A
(13)
A.1 ORTHOGONALITY AND BLOSED-FORMA SOLUTION
Strictly speaking, Section 4.1 requires A to an orthogonal matrix for the necessary equivalence change of basis. In practice, we find our appearance
A A 'Z to hold for the interpretation factor matrix A E CCxC to very close to orthogonal. For example, for BigGAN on the 'alp' class layer 7, we find the mean element of T A Iel be 8.33e-4, We show in Fig. 7 the 1st appearance vector localized in the image (following the procedure in Section 4. 4.1.2) through both A A Z and A 'Z, where we see that, for the purposes of understanding the appearance factor visually as controlling the 'sky" concept, the two results are ncar-identical. Thus, we can take A to be an orthogonal matrix for our practical purposes.
If a more strict form of orthogonality is desired, one has the option to instead compute the appearance factors' solution in closed-fis form following Xu et (2005). In particular, let Z E IR M xcxS be the batch of mode-3-unfolded feature maps in the main paper. Using the mode-n product (Kolda & Bader, 2009), the partial multilinear projection the feature maps onto the parts subspace is given by ) = Z PT E TR M MMC Rs. The appearance factors' solution is then given in closed-fomm by the leading cigenvectors of Y (2)
CCCC (Xu et al., 2005). When using this closed-forn foom solution,
(2)
find the mean element of A ATA Icl is 9 in the same setting above.
14

---

Published as a conference paper at ICLR 2023
Coordinates of 1st concept
Coordinates of 1st concept (A-1Z))
Original image
(A Z)1
7.5
5.0
5
2.5
0.0
-2.5
-5.0
-7.5
Figure 7: Comparing the concept localization with A and A A1 we see that they are near-identical.
B ABLATION STUDIES
In this section, we present a thorough study of the various parts of our method, and the resulting learnt parts factors.
Part
MPCA (separable)
MPCA
Ours (constraited)) (unconsirained)
Ours
Figure 8: Ablation study comparing the parts factors learnt with various constraints and formulations. As can seen, only our constrained formulation learns factors that span local parts-base based semantic regions.
B.0.1 CONSTRAINTS AND FORM
We first study the impact of the non-negatyvity constraints on the parts factors, and the importance of operating the mode-3 unfolded Zi(3) E H--H-W tensors (rather than their original 3rd 3rd-order form Z, HRHxxxx Wx).. We show along the rows of Fig. 8 the resulting parts factors using various forms decomposition and constraints. In particular, naively applying MPCA (Lu et al., 2008) (row 1) decompose Z, imposes a separable structure between the spatial modes, restricting its ability to capture semantic spatial regions. Moreover, even when combining the spatial modes and decomposing Z(3)) the solution given by MPCA (Lu et al., 2008) (row 2) and by optimizing our method without any non non-negane rennetivvvityy constraints (row 3) leads to parts factors spanning the entire spatial window. This is due to the non aoddiiiiivee nature of the parts. However, shown row 4 of Fig. only our constrained method successfully finds local, non-overlapping semantic regions of interest.
B.0.2 PARTS FACTORS REFINEMENT
Finally, we showcase the benefit our optional parts factors refinement process for data with no alignment. In row of Fig. 9, show the global parts factors overlaid over the target samples.
15

---

Published as a conference paper at ICLR 2023
Clearly, for extreme poses (or in the of data with alignment, such with animals and cars), these global parts will not correspond perfectly to the specific sample's 's parts (row Fig. 9). However, after few projected gradient descent steps of Eq. (6), the refined parts factors span the specific parts of the individual samples more successfully. This refinement step is very fast; for example, at 6 it takes only 127ms (100 iterations).
(row of Fig. 9) that
Input
Global
Refined
Figure 9: Visualization of the global parts factors (middle row) and the refined factors (bottom row)
for particular samples (top row).
C ADDITIONAL EXPERIMENTAL RESULTS
In this section. we provide additional experimental results to support the main paper. We begin by presenting additional qualitative results local image editing in Appendix C.1 and Appendix C.3. After this, we present additional ablation studies in Appendix C.6, and Appendices C.7.1 and C.7.2. Finally, we showcase additional experiments analyzing the runtime in Appendix C.8. However, first provide more detailed overview figure for our method-in Fig. 10 we show in more detail our proposed semi-nonnegative factorization in the context of a pre-trained image generator.
C.I TEST-TIME MANUAL PART SELECTION
In the main paper, describe how pixel-level control is facilitated by our method in way that is not possible with the SOTA (Zhu et al., 2021a; 2022). To demonstrate this, we show in Fig. 11 the resulting image when we add an appearance vector controlling the eye size to our spatial part for the eye region manually edited to cover over only one of the eyes. As we see, this clearly enlarges just single eye, leaving the other untouched.
C.2 BACKGROUND REMOVAL
Whilst not the main focus of the paper, show that one can use the appearance vectors identify the background of an image as one way of further demonstrating the quality of the concepts learnt and their utility for additional tasks. Prior work (Voynov & Babenko, 2020; Keris-Kyriazi al., 2022) identify "background removal" interpretable direction in the latent space of BigGAN (and GANs more generally the case of Melas Deylas- al. (2022)). By using our thresholded saliency maps for the "background" concept as mask at the pixel-level, we can straightforwardly perform background removal.
For BigGAN, we show in Fig. 14a comparison to Voynov et al. (2020) on their custom generator weights. For StyleGAN2, we make both a qualitative (Fig. 14b) and quantitative (Fig. 14c) to the mask predictions generated by the recent SOTA Melas-Kyriazi et al. (2022) on 4 different datasets. As can be seen, our method performs extremely well under the loU metric, despite not being trained for this task. We generate grouund-truth' background/foeerround masks using the general-purpos U model Qin et al. (2020), which whilst not perfect, is often remarkably accurate as can be seen from the qualitative results.
16

---

Published as a conference paper at ICLR 2023
Tensor factorization
Appearance a
a
factors
Multiplicative
Rc
interactions
Parts
Aer P1
Air2 P2
Air8
P8
factors
Learnt parts
Fake image
Z3 Rd
GLay
Gu
H
Z
Latent code
W
Intermediate
feature maps
Pre-trained image generator
Figure 10: A more detailed overview of our method, in the context the image generator. We apply semi- noneegooeee tensor decomposition the intermediate convolutional feature maps, to extract factors for parts and appearances.
Original
Edited
Manipulated part
One can manually edit the learnt parts factors or provide custom ones for pixel-level control. For example, using half of the "eyes" part to affect only single eye.
Figure 11:
C.3 ADDITIONAL QUALITATIVE RESULTS
Here we show many more qualitative results, on 5 generators. Results for StyleGAN2 are shown in Fig. 12, and the other models in Fig. 13. As can be seen, our method is applicable to large number of generators and diverse datasets. Additionally, we show in Fig. 15 that our method learns a diverse range of appearance vectors. Thus, it is possible to chain together sequence of local edits, removing some objects, and modifying the appearance of other regions. This results in powerful technique make complex changes scene of interest.
with which
C.4 ADDITIONAL COMPARISONS
We next include additional comparisons in Fig. 16 to the SOTA methods for all 4 local edits found in all baseline methods. As can be seen, whilst the SOTA local image editing methods excel at making prominent, photo-realistie changes to the ROI, they once again affect large changes to the image beyond just the ROI, visualized by the MSE in the bottom rows. This is quantified once again with
17

---

Published as a conference paper at ICLR 2023
Original
Part(s)
Modified appearance
Original Part(s)
Modified appearance
StyleGAN2
Figure 12: Additional local image editing results for StyleGAN2: each column, we edit the target part with a different appearance vector.
18

---

Published as a conference paper at ICLR 2023
Original
Part(s)
Modified appearance
Original
Part(s)
Modified appearance
BigGAN
StyleGAN3
StyleGAN1
ProgGAN
Additional local image editing results for 4 generators: each column, edit the target part with different appearance vector.
Figure 13:
the ROIR metric for six more local edits found in both our method and StyleSpace (Wu al., 2021) in Table 3. We also show these images' quality is comparable to the SOTA by computing the FID (Heusel al., 2017) metric Table for 10k samples per edit.
Table 3: Additional ROIR (L) results for six more local edits (Ik images per edit)
Wide nose
Dark eycbrows
Light eycbrows
Glance left
Glance right
Short cyes.
StyleSpace
2.52 + 0.91
1.99 + 1.28
1.95 1.13
1.54 + 1.14
1.50 + 1.17
1.91 + 1.41
Ours
0.80 + 0.23
0.51 = 0.19
0.41 0.13
0.68 + 0.24
0.67 + 0.24
0.46 + 0.13
C.5 SALIENCY MAPS
In this section show additional results saliency map generation. In Fig. 18 show this for the "background" concept, which can be used to remove the background in the image, with the negative mask. We also provide localization results additional concepts such as "sky" and "concrete" in
19

---

Published as a conference paper at ICLR 2023
Original
UDID
Ours
Original
UDID
Ours
Original
UDID
Ours
Mask
Mask
Mask
Edited
Edited
Edited
(a) A comparison of our method to Voynov & Babenko (2020) for background removal and object detection:
leave the object interest largely untouched.
Original
GT
Un-Seg
Masked
Ours
GT
Un-Seg
Masks
Ours
(b) Qualitative comparison on StyleGAN2 to Melas Melas-Kyriazi et al. (2022) for AFHQ-dogs. The single datasct-
specific background appearance factor often accurately segment the images.
AFHQ-dogs
AFHQ-cats
FFHQ
MetFaces
Meun IoU
Median IoUl
Meun loU
Median IoU
Meun IoU
Median IoU
Meun Tol."
Median IoU
Melas Kyriuzi al. (2022)
0.48 1 0.22
0.52
0.34 & 0.18
0.30
0.37 1 0.23
0.36
0.28 1 0.27
0.21
Ours
0.65 J. 0.19
0.72
0.54 :: 0.19
0.58
0.46 1. 0.19
0.47
0,52 :: 0.19
0.53
(c) Quantitative comparison Melas Mclas-Kyriazi al. (2022) for StyleGAN2
datasets, using the author's 0.2. We train for the default 300 iterations on datasets apart from the
official code, and recommended two face-based datasets, where we find their method to give much better results at 100 iterations. For cach dataset we compute the IoU over 1k samples.
Figure 14: Demonstrating the appearance factor's semantics through background mask generation.
20

---

Published as a conference paper at ICLR 2023
Learnt parts
+oap P6
+QazPi T
+0a2 Po
T
+0a.p7
spire
+ #-trees
clouds
windows
shutter
strckk
shutter ee ttcck
shuters
shucers
shuters
Original
Edited images
Figure 15: Progressively editing the feature maps with appearance vectors at various regions. Ulti- mately one can perform complex composite changes to a scene.
Table 4: FID (4) (Heusel et al., 2017) on 10k FFHQ samples per local edit.
Eyes
Nose
Open mouth
Smile
GANSpace (Härkönen et al., 2020)
29.92
30.60
46.85
30.62
SeFa (Shen & Zhou, 2021)
30.23
31.52
31.73
29.53
StyleSpace (Wu et al., 2021)
29.81
29.62
31.47
30.19
LowRankGAN (Zhu et al., 2021a)
30.57
29.18
30.95
30.60
ReSeFa (Zhu et al., 2022)
29.37
30.02
29.63
29.45
Ours
29.91
29.77
29.26
28.91
Fig. 20. Interestingly, our method learns concepts at layers depth BigGAN. For example, we foreground, mid-ground, and background concepts emerge independently visualized in Fig. 19.
C.6 DECOMPOSITION RANK
We next demonstrate the impact of the choice of rank Rs for the parts factors. Shown Fig. 21 are various global parts overlaid over a random image from the dataset. As can be seen, smaller value of Ry leads to larger semantic parts such those spanning the entire face, whilst a larger value of Rs leads to spatially smaller parts such as the eyebrows and teeth. This can in essence be viewed as hyperparameter that affords a user control over the size of the parts one wishes to learn. Additional comparisons shown for many datasets in Figs. 22 to 24.
C.7 INITIALIZATION STRATEGIES
C.7.1 PARTS FACTORS INITIALIZATION
We find our method is robust to different non non-negative initialisations of the parts factors. This is crucial benefit of our method-we we do not require an SVD-based initialisation (Cichocki al., 2009; Boutsidis & Gallopoulos, 2008; Yuan et al., 2009) for the parts factors, and thus need not solve the corresponding 5-dimensional eigenproblem in this step. As case point, at layer in an 18-layer StyleGAN (Karras et al., 2019), an (0334-diimnsional eigenproblem must be solved solely for the parts factors initialization thus going deeper quickly becomes infeasible without the flexibility of the proposed method. Instead, our formulation permits initialization through sampling each element of the parts factors from a random uniform distribution on the interval [0, 0.01]. This allows us to perform decomposition at much later layers in the network than would otherwise be possible, and
21

---

Published as a conference paper at ICLR 2023
Orginall
Orggnal
GANS
Safa
Syddppce
LawfarkGAN
BeSefa
Oure
CANSpacc
Safa
StrkSpocc
LewtarkAA
Reinfa Ours
Edits
Eyes
MSE
Edits
Nose
MSE
Edits
Smile
MSE
Edits
Mouth
MSE
Figure 16:
Qualitative comparison SOTA methods for 4 local edits. As can be seen, method succeeds in keeping the area outside the ROI unchanged, as intended.
consequently one can discover more fine- lineggrined appearance vectors (due to the way in which later layers in StyleGAN control more fine- grainer styles (Karras et al., 2018)).
C.7.2 CHANNEL FACTOR INITIALIZATION
We find the leading left-singular vectors of the channel mode's scatter matrix contain particularly semantically meaningful directions. This initialization is what allows us to locate common concepts such "background" with ease, due to the ordering of the singular vectors by their singular values.
Intuitively, the leading few left- singula vectors for the channel mode's scatter matrix capture frequently occurring appearances and textures. For faces, this largely textures such as skin, and hair. Whilst for other datasets, common textures include the sky or floor. We show additional examples of this in Fig. 25, where the first columns of the appearance basis (through our initialization) correspond to the "background" and "skin" texture. For example, by adding this appearance vector, seen in Fig. 25 (b). Whilst we find our decomposition also works with random initialization of the channel factors, the first few vectors do not necessarily correspond to the more frequently appearing concepts in the same way the SVD-based initialization provides, meaning they less easy to interpret.
remove the facial features of person
C.8 RUNTIME AND OBJECTIVE
An important benefit of our method is the lack of need to compute expensive gradient maps or Jacobians with respect to target regions, as is required in Zhu et al. (2021a); Wu et al. (2021); Zhu et al. (2022). To quantify this, plot in Fig. 26 the total training time required to train the SOTA methods to produce the four local directions used in the main paper. particular, we train the methods for all regions of interest ("mouth". "eyes", and "nose"), with single Quadro RTX 6000
22

---

Published as a conference paper at ICLR 2023
Original
Original
Original
Original
StyleSpace
Ours
Style eSace
Ours
StyleSpace
Ours
Sty eSpace
Ours
Dark eyebrows
Edits
Glance right
MSE
Light eyebrows
Edits
Glance left
MSE
Figure Qualitative comparison more local edits for the StyleSpace method.
layer: 8, appearance: 0
Projection Original
Mask
Masked
StyleGAN2 (AFHQ dog)
Figure 18: Saliency maps, generated masks, and subsequently removed backgrounds on AFHQ Dog. with StyleGAN2.
GPU, using the official authors' 2odebases We find that method takes less than 1 1/400th of the training time LowRankGAN (Zhu et al., 202121 and 1/170th the time of StyleSpace (Wu et al., 202l)-greatly speeding up the task of local image editing.
Alternatively, one can use an optimizer such as Adam (Kingma & Ba, 2014) in an autograd framework (e.g., PyTorch (Paszke et al., 2019)) compute Algorithm 1. We find this removes some sensitivity
"StyleSpace: https: /ttpttb.bbb/.gggggggg/gttgtggh
Low NowRanGGAN: https: htt//gtttttt thub. com/ zhuji apeng/ owRankGAN, ReSeFa: https: https:://ithhbb.cm//// wtwp:/ttthoooooooooo/otooootatie apeng/ resefa
23

---

Published as a conference paper at ICLR 2023
layer: 9, appearance:
layer: 9, appearance:
Original
Projection
Mask
BigGAN (breakwater)
BigGAN (alp)
layer: 9, appearance:
layer: 9, appearance: 2
Original
Mask Projection
BigGAN (alp)
BigGAN (alp)
Figure 19: Saliency maps and generated masks on BigGAN "alp" and "breakwater". We see concepts different levels of depth emerge.
the learning rate that comes with vanilla gradient descent. However, find that when solving our objective manually with the gradients in Eq. (12), our method takes less than 1/3rd of the time to train. This particularly useful performance boost when decomposing later layers in the network.
C.8.1 RUNTIME ABLATIONS
To isolate the impact of the appearance factor training, in Fig. 27 we show the importance of descending the gradient in the channel subproblem (after 1000 initial steps of the parts subproblem) on top the HOSVD initialisation, for 1000 samples. As can be seen, the cost strictly decreases as function of the number of iterations T, far below that of the HOSVD initialisation.
We also show in Fig. 28 the impact of the number of training iterations for various choices of rank Rs. As can be seen, the model is relatively stable over iterations in assigning global parts factors.
C.9 IMPLEMENTATION DETAILS
We use modified version of the GenForce (Shen et al., 2020b) library for the ProgressiveGAN, StyleGANI, and StyleGAN2 models We also use the TensorLy (Kossaifi et al., 2019) library for
in addition to StyleGAN3: http://ggghhhhccoo///////ttt//a/ BigGAN: http:/::////.it..oo...//////a/aa http:///wwwoggggggoaaaaaa/toeere
gGAN.
24

---

Published as a conference paper at ICLR 2023
layer: 8, appearance: 1
Projection Original
Mask
layer: 9, appearance: 1
Mask Projection Original
shuf
StyleGAN2 (Church)
Figure 20: Saliency maps and generated masks on StyleGAN2 (church), Shown here what one could deduce to be 'sky' and 'concrete' concepts.
Rs =
Rs 8
Rs 10
Rs 16
Rs 20
Figure 21: Learnt global parts factors for varying ranks Rs, with StyleGAN2 trained on FFHQ at layer = 9, for 2000 iterations. As can be seen, shared semantic regions such as the eyes, the nose, and the mouth are captured by the parts factors.
the autograd implementation of our method. In practice, we find it useful (in terms of run-time) to descend the gradients of our loss function stochastically, sampling new activations each iteration.
C.9.1 BASELINES
For both the quantitative and qualitative results for the baseline methods, we use the following directions annotated from the pre-trained models by the authors, where available:
GANSpace (Härkönen et al., 2020): we use the following Sye_OOpenees, Nose_ length, Screami ng, and Smi ee.
directions:
25

---

Published as a conference paper at ICLR 2023
Global spatial factors (StyleGAN2, AFHQ-cat)
Global spatial factors (StyleGAN2, AFHQ-cat)
Name
Ra 20
Rs 16
Ram 20
RemB
Ry 10
Rg 16
Ry 20
Rg Ra- 302
16
1k..2 20
Ra-is
Fi,
R4 20
II
20
8
II
II
Figure 22: Learnt global parts factors for varying ranks Rs, with StyleGAN2 trained on two separate splits of AFHQ at various layers I. (zoom for detail). As can be seen, common semantic regions such as the eyes, the nose, and the mouth are captured by the parts factors.
Global spatial factors (StyleGAN1, CelebA-HQ)
Rs
Rs
Rs 10
Rs 16
Rs 20
1
Rs = 4
Rs =
Rş = 10
R5 = 16
Rş = 20
6
1
Rs
Rs
Rs 10
Rs 16
Rs 20
Figure 23: Learnt global parts factors for varying ranks Rs. with StyleGANI trained on CelebA- -HQ various layers I. As can be seen, common semantic regions such the eyes, the nose, and the mouth are captured by the parts factors.
26

---

Published as a conference paper at ICLR 2023
Global spatial factors (StyleGAN2, FFHQ)
Rs
Rs = 8
Rs 10
Rs =
Rs = 20
1-6 1
Rs =4 4
Rs 8
Rs = 10
Rs = 16
Rs = 20
Rs
Rs = 8
Rs = 10
Rs = 16
Rs = 20
1.7 1-7
Rs 4
Rs 8
Rs = 10
Rs = 16
Rs = 20
1-8 8
Rs 4
Rs 8 8
Rs = 10
Rs = 16
Rs = 20
1
Figure 24: Learnt global parts factors for varying ranks Rg, with StyleGAN2 trained FFHQ various layers l. As can be seen, common semantic regions such as the eyes, the nose, and the mouth are captured by the parts factors.
SeFA (Shen & Zhou, 2021): we use directions manually found by ourselves that most closely resemble the target attributes.
LowRankGAN (Zhu et al., 2021a): we use the following
directions:
expression, eye_size, mouth_open, nose.
StyleSpace (Wu et al., 2021): we use following author-annotated directions: 8_289, $_113,6 202, and 14_239, where x_y is channel generator level y as described in (Wu et al., 2021).
ReSeFa (Zhu et al., 2022): we use the following author-annotated directions: eyesize, mouth, and nose_length. We manually find direction that most closely resembles smile.
27

---

Published as a conference paper at ICLR 2023
Parts factors
Parts factors
Parts factors
Original
Original
Original
Edited
Edited
Edited
(a) ProgressiveGAN
(b) StyleGAN2
Figure 25: The appearance vectors obtained via an SVD-based initialization are particularly seman- tically meaningful: for example, the 1st column in the basis of PGGAN removes objects from the image in CelebA-HQ, whilst the 2nd column for StyleGAN2 adds skin in FFHQ.
Training time (minutes)
400
52.78
0.00
347.79
271.43
300
0.00
200
177.12
GPU minutes
100
0.00 0.87
CPU minutes
0
LRG
StyleSpace ReSeFa
Ours
Figure 26: Total training time (to train the models used for the quantitative results) for method and the SOTA, using a Quadro RTX 6000.
28

---

Published as a conference paper at ICLR 2023
1e4
Rs 4
1e4
Rs =
Reconstruction loss
4.96
3.70
4.95
3.68
3.66
4.94
3.64
0
50
100
0
50
100
le4
Rs 16
1e4
Rs = 32
Reconstruction loss
2.425
HOSVD init.
HOSVD init. train
2.400
1.35
2.375
2.350
1.30
0
50 Number of iterations T
100
0
50
100
Number of iterations T
Figure Appearance factor subproblem when using the fixed HOSVD initialization VS additionally descending the gradient, as function of 'I' and for various number of parts Rs.
T T = 10
T' 20
T' 100
Rs 4 Rs
Rs = 16 Rs
OO
Figure 28: The learnt global parts factors function of training iterations T for various choices of rank Rs: the parts factors are stable over iterations and reliably correspond to semantic parts across different values of Rs--
29


Reviews:
[{"review_id": "vyysC2eQn7", "reviewer": "Reviewer_ixJz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a factorization of features maps in GANs such that spatial location is disentangled from appearance. Using NMF, the methods obtains a set of appearance prototypes and a set of non-negative spatial activations which, when combined with instance specific coefficients, reconstruct the feature map. Results show that this obtains localized compact regions corresponding to parts which are uncovered without supervision. These parts can be used to perform unsupervised object detection or even local editing by change the appearance prototype that is used at that location.", "summary_of_the_review": "The idea is very sound and has practical application. There are a few question as to why the extra mile was not made in order to have a much more intuitive method.\n\n\n-----\n\nThe rebuttal discuss most of my concerns which were more remarks and discussion material than actual requests for change.", "strength_and_weaknesses": "This is a very interesting paper that follows recent work on analyzing what happens in the feature maps of GANs. The idea of decomposing the feature map using an additive process is very sound. The results are also very good: parts are clear and the editing are OK.\n\nThat being said, there are some questions about this work:\n1. The process is not entirely additive: the coefficient in $\\Lambda$ can be negative, which means that some appearance and/or some spatial locations can be removed. Why not just making this non-negative as well? Sure it renders the optimization problem more complex, but it also corresponds more to the intuition given as motivation.\n2. $\\Lambda$ is arbitrarily set as $A^\\top Z_i P$. Why? Why not optimize $\\Lambda_i$ for each image? In classical NMF, instance coefficient are usually optimized along the atoms in the dictionary. And since you perform a few step of instance specific optimization for $P$ anyway, why not optimize it also.\n3. In the editing, a rank 1 element is added to the original feature map. Why not removal as well? Why not reducing the weights of the appearance vectors that where at the spatial location as well?\n4. Global spatial decomposition is problematic because it assumes a fixed layout, hence the instance specific optimization. Did you try a translation invariant formulation by, e.g., using a convolution operator in the decomposition? (this is very successful in NMF)\n5. In 4.1.1, it is stated \"frequently learns an appearance vector for a high-level background concept\". Only one? So a single vector encodes all the possible backgrounds of the generator? That sounds implausible.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MGas3DBzJwD", "reviewer": "Reviewer_zd35", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to decompose the feature maps of a GAN generator into spatial and semantic components. This decomposition allows editing the output image at the level of parts. The experiments show that the method can be applied to several different GAN architectures and datasets. Additionally, due to the simplicity of the approach, the method is very fast.", "summary_of_the_review": "Overall the paper presents a simple approach to enable more fine-grained editing of GAN images by decomposing feature maps. Despite the strengths, the current state of the evaluation makes it difficult to assess the benefits of this method over existing techniques. ", "strength_and_weaknesses": "## Strengths\nThe method is very fast and simple as it is based on a linear decomposition of feature maps of the generator. This means that no network training is needed to discover the parts and appearances. \n\nIt is possible to finetune the decomposition for individual images to allow for more precise edits. Additionally, as the decomposition is spatial, the edits can be more targeted that editing the latent space of the GAN alone. \n\nThe method has been evaluated on a range of GAN architectures and datasets which shows the generality of the method and its independence from specific generators.\n\n## Weaknesses\n\nA main weakness of the paper is the evaluation. Although evaluating semantic editing of GAN generated images is a difficult problem, it is necessary to support the claims of the paper.\n### W1 Semanticity\nIt is not clear if the part decomposition is indeed semantic and not spatial. It is likely that the linear decomposition prefers decomposing into spatial regions and relies on the alignment of samples across the dataset. See e.g. Fig 9 columns 4,7,8 where instance alignment is not present. For conditional GANs, can semantic parts be derives across classes to find correspondences between parts (e.g. to replace dog legs with cat legs)?\n\n### W2 Background\nThe paper emphasizes the ability of the model to identify the background/foreground of images. The fact that this is possible has been shown not only by (Voynov&Babenko 2020) but also in (Melas-Kyriazi et al, 2021) for a large variety of generative models. The quality of the background discovery could be measured possibly in two ways. Either through inversion of the GAN on real images with associated ground truth on a saliency segmentation dataset or relative to a supervised model, directly on the generated images. This would be helpful, as currently the results look more similar to a simple spatial decomposition rather than a semantic one. (e.g. c,d in Fig 4, Fig 19) similar to the decomposition on face datasets seemingly strongly relying on the alignment between images. \n\n### W3 Hyper-parameter and ablation\nThe method contains several hyper-parameters and specific choices. It would be valuable to understand their influence on the model. E.g.: How well does the model work just with SVD alone? What is the influence of the number of iterations T? Are there substantial differences between different GANs trained on the same data - do certain architectures lead to feature maps that are easier to decompose? Can appearance vectors of the same part be swapped between instances? How does the layer choice matter? Does the user need to select the layer for decomposition or is there an automated way to choose? Other forms of clustering could be compared, e.g. k-means.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9Uaa2sur5L9", "reviewer": "Reviewer_WFbU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a way to manipulate intermediate latent features of pre-trained generative models to edit local parts of the image.\nSpecifically, they decompose a tensor of sampled latent features into two tensors, in which one could be interpreted as the set of appearance features and the other as the set of saliency maps that represent how much a specific appearance feature spans over the pixel-level space (coined as parts feature).\nThis is induced by constraining the decomposition procedure to make the parts feature nonnegative.\nThe decomposed features enable the manipulation of an original image by adjusting the attribute feature for the specific parts feature of interest.\nAs suggested by the results, the proposed method can be applied well to various pre-trained generative models with different architectures and training datasets.\n", "summary_of_the_review": "This paper tackles an important problem which is helpful for both interpreting and further utilizing pre-trained generative models.\nThe proposed method is clear and novel, and its efficacy is well proved with various experimental results.\n", "strength_and_weaknesses": "Strengths\n- Dissecting a pre-trained generative model is an important and interesting task, and proposed method is novel and strong enough both in terms of its formulation and the quality of manipulation results.\n- The goal of strict localized editing is well demonstrated qualitatively, and also quantitative evaluated with an appropriate evaluation metric.\n- The authors present experimental results, and the changing trend in the decomposed features according to the hyperparameters’ change vastly in the supplementary material.\n\nWeaknesses\n- This method would only work for the models that are trained on images of single object type.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-2SYeLQ7NZL", "reviewer": "Reviewer_3WTy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a novel way of performing localized edits in the latent space of convolutional GANs like. The authors propose first to identify visual concepts by factorizing network activations at a certain layer into a set of appearance factors applied at certain spatial locations encoded by part factors. Once the factorization has been computed the learned appearance factors act as templates of visual concepts and can be used to identify parts of the generated image corresponding to a high level semantic concepts like “background” or “nose”. By swapping features in the latent space according to the localization provided by the appearance factors it is possible to apply local edits to images. The method is fast, architecture agnostic and by design limits the edits to only a user specified small portion of the image.", "summary_of_the_review": "The idea behind the work is not novel, but the formulation is and the results are quite convincing. The method has some limitations but overall I feel like the authors made a good job at proving their contribution experimentally and providing a detailed analysis of the method through ablation studies. The main weakness of the paper is that it makes some strong assumptions in the factorization of the activations (weakness b). Lifting these would make the case for a stronger submission.", "strength_and_weaknesses": "## Strengths\n\n+ The proposed method does not make any assumption on the architecture of the GAN (except for the generator to be convolutional). As such it can be applied to a vast array of existing models without requiring modifications.\n\n+ The completely unsupervised nature of the formulation, together with the fast convergence time makes it very flexible and easy to be applied to many different generators and datasets (the authors tried it on 5 generators trained on 5 different datasets with good results).\n\n## Weaknesses\n\n1. The proposed method is remarkably similar to previous works in the literature (Collins 2020), with the main difference being the method used to cluster activations of the neural network. While Collins used k-Means, the authors proposed to use matrix factorization to identify patterns in the feature space. From my perspective the two approaches achieve a very similar outcome with a slightly different formulation. The authors of this work however explore the application of this clustering discovery more deeply than what was done in [Collins 2020]. I don’t agree with the point raised by the authors at the end of the related work section where they say that “Collins is limited to part swapping”, while their method is not. From my perspective their method is also doing part swapping but allows the extra flexibility of either swapping features between two images or using one of the learnt appearance factors. \n\n2. The proposed method is completely unsupervised and task agnostic and as such can be applied out of the box to many different models and datasets. However the formulation does come with some hyperparameter that drastically can change the outcome: mostly the number of factors to extract and the layer of the generator to factorize. The ablation study shows that these two components, as expected, have a big impact on the factorization extracted and as such will need to be retuned for a new task and generator. Moreover once the factorization has been computed there’s still the need of some “human supervision” to map the learned factors to semantic concepts. As such this method might in the end require more human labor for practical application than competing works relying on pre-trained classifiers.\n\n3. The proposed formulation implicitly assumes that the images are aligned while factorizing activations to compute the activations factors. While the authors show that the part factors can be aligned to a specific image with an ad hoc optimization, this is something done only in a second phase but not during the main optimization. From my perspective there’s no reason (except the computation cost) to not model separate part factors for each one of the activation maps used during the main factorization process. This will make the method more robust to datasets which are not very well aligned. With the current formulation the method cannot explain features that are not very well aligned across different samples. \nA question I have for the authors is if there is anything preventing the use of a per sample part factorization during the main optimization?\n\n4. The fact that the method performs very localized edits compared with the competitors can be seen both as a pro and as a cons. In particular it might be a limitation in the sense that it cannot adapt the full image to the particular edit being introduced. A practical example of this could be removing an object from an image and replacing it with background, but not removing the shadow casted by it. Some discussion about the limitations of the method along this direction will make the paper stronger.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster