Finding an Unsupervised Image Segmenter in each of your Deep Generative Models
Authors: Luke Melas-Kyriazi
Christian Rupprecht
Iro Laina
Andrea Vedaldi
Venue: ICLR 2022 Poster (2022)
Keywords: unsupervised
generative
deep learning
segmentation
object segmentation

Abstract: Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs. In this paper, we develop an automatic procedure for finding directions that lead to foreground-background image separation, and we use these directions to train an image segmentation model without human supervision. Our method is generator-agnostic, producing strong segmentation results with a wide range of different GAN architectures. Furthermore, by leveraging GANs pretrained on large datasets such as ImageNet, we are able to segment images from a range of domains without further training or finetuning. Evaluating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data. Broadly, our results demonstrate that automatically extracting foreground-background structure from pretrained deep generative models can serve as a remarkably effective substitute for human supervision.

Published as a conference paper at ICLR 2022
FINDING AN UNSUPERVISED IMAGE SEGMENTER IN EACH OF YOUR DEEP GENERATIVE MODELS
Luke Melas-Kyriazi
Christian Rupprecht
Iro Laina
Andrea Vedaldi
University of Oxford [lukemk, chrisr, iro, aoobott ox. a...ac.oo
ABSTRACT
Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs. In this paper, develop an automatic procedure for finding directions that lead to breground-bbcckrouudd image separation, and we these directions to train an image segmentation model without human super- vision. Our method is generator-agnostic, producing strong segmentation results with wide range of different GAN architectures. Furthermore, by leveraging GANs pretrained on large datasets such as ImageNet, we are able to segment images from range of domains without further training or finetuning. Eval- uating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data. Broadly, our results demonstrate that automatically extracting foreground- background structure from pretrained deep generative models can serve as re- markably effective substitute for human supervision.
INTRODUCTION
Recent years have seen rapid progress in the field of deep generative modeling of images, driven by a proliferation of research into Generative Adversaial Networks (GANs) (Goodfellow et al., (014). Nowadays, it is possible to generate high-resolution images of realistic objects and scenes (Krras et al. 2019 Brock al., 2019). However, with the exception of generation for computer graphics, there has been limited research into how might be able to leverage the representations learned by these powerful generative models to enhance other tasks, particularly those involving semantic understanding (e.g. classification segmentation).
Generally, deep generative models learn to map latent codes to images, imposing simple statistical structures the distribution of the latent codes, such assuming i.i.d. Gaussian distribution. Due to this structure, in some cases code dimensions acquire specific meanings which can be related to human-interprettble concepts (e.g., the rotation size of object); however, the code space in high--gguuhe high-quality generators (c.g.. BigGAN (Brock et al.) 2019). BigBiGANs (Donahue & Simonyan, 2019). StyleGAN (Karras al. 2019) is usually not easily interpretable. Nonetheless, is intuitive that an efficient generative process should account for the structure of natural images, including for example the fact that images often comprise distinct foreground and background regions.
In this paper, we validate this hypothesis by learning to separate foreground and background image regions from generator networks. Our approach starts from an arbitrary, off of-thhe-sheff high-quality generator network trained on large corpus of (unlabeled) images. While these generator networks are not explicitly trained for foreground/backkronn emerges implicitly
segmentation, we show that such separation step efficiently encode callisicalllyllooiin images. Specifically, de- sign a probing scheme that can extract such Ooreground/backgrron information automatically, i.c without manual supervision, from the generated images.
This scheme works as follows (cf. Fig. We start from a random code in latent space and learn fixed, global offset that results in change in the generated images. The offset is learned alter the appearance of foreground and background such that a mask can be extracted from the changes in image space.

---

Published as a conference paper at ICLR 2022
Crufts
Figure I: We automatically find a universal latent direction in a GAN that can separate the foreground from the background without any supervision. We can then generate unlimited samples with masks to train a seg- mentation network that achieves state-of-the-art unsupervised segmentation performance.
The resulting masks provide segmentation maps for the generated images, but they cannot yet be used to segment images from the real world. Given a natural image, the obvious approach would be find the corresponding code in the latent space of the generator, and then obtain mask with our method. Unfortunately, this inversion process is less than trivial. In fact, recent work provides strong evidence that the expressiveness of GANs is insufficient to encode arbitrary images (Bau ct al. 2019), meaning that the inversion problem has solution in general.
As we aim to build a general-purpose segmentation method, we take a different approach: we gen- erate a labelled image dataset with foreground/backgrounn segmentations and use the generated dataset to train a standard segmentation network. With this, we show that our method can suc- cessfully learn accurate foreground-buckgroundd segmentation networks with manually provided labels at all. Differently from prior work in GAN-based image segmentation, we neither design new GAN architecture specifically for the task of segmentation extract segmentation information from an existing GAN. Thus, we can discover meaningful latent directions for any GAN with no need for model-specific manual intervention.
use manual supervision to
Extensive experiments on five segmentation datasets across twelve different GANs demonstrate the effectiveness and generalizability of our approach. Moreover, by constructing our image segmenters from generator networks trained on a generic large-scale datasets such as ImageNet, our method can learn to generically segment objects from wide range of visual domains. Specifically, when we apply our generic segmenters to the CUB200 (Welinder et al. (2110) and Oxford Flowers (Nilsback) 2009) datasets, we attain very strong wnrground-backgrown data. Similarly, when we apply our generic segmenters to three saliency detection benchmarks, our method approaches and sometimes even exceeds the performance of supervised and handcrafted saliency detection methods. An analysis of our results also shows that segmentation performance directly correlates with the quality of the underlying GAN, suggesting that foreground//acckrround separation is an important concept in learning generative models.
results despite not training on this
Finally, we demonstrate that our method may be used as a drop-in replacement for saliency networks for the purpose of learning pixel-wise semantic image representations. These pixel-wise represen- tations can then be clustered to obtain fully unsupervised semantic segmentations, extending our method beyond boreground-backgroune time that it is possible to obtain semantic segmentations using GANs, but also that this information may be extracted from a wide range of generic GANs trained on general-purpose image datasets.
segmentation. Thus, we not only demonstrate for the first
2 RELATED WORK
Below, we describe how our method relates to recent work in interpreting generative models and object segmentation. Due to space constraints, we include additional related work in the Appendix.
Interpreting Deep Generative Models. Several works have proposed methods for decomposing the latent space of a generative model into interpretable or disentangled directions. Early work included Beta- VAE (Higgins et al. 2017), which modified the variational ELBO in the original VAE formulation, and InfoGAN (Chen et al. 2016), which maximized the mutual information between subset of the latent code and the generated data. Later work has sought to disentangle factors of variation by mixing latent codes (Hu et al. 2018), adding additional adversarial losses (Mathieu et al. 2016), and using contrastive learning (Ren et al. 2021
2

---

Published as a conference paper at ICLR 2022
Our work follows recent line of research that looks for structure in large, pretrained generative models. [Shen& & Zhou (2020) perform a direct decomposition of model weights to find disentangled directions, Peebles al. (2020) penalize nonzero second-order interactions between different latent dimensions, and Voynov & Babenko (2020) find interpretable directions by introducing an additional reconstruction network.
Differently from the works above, we conduct a deep study of one specific type of structure (fore-
ground/background separation) encoded in the latent space. Other works have taken this approach in the context of extracting 3D structure from 2D images; for example, IG-GAN (Lunz et al. 2020) uses neural renderer to recover 3D (voxel-based) representations of scenes, and GAN2Shape (Pan et al. 2021 exploits viewpoint and lighting variations in generated images to recover 3D shapes.
Unsupervised Object Segmentation. Prior work on unsupervised object segmentation can be divided into two categories: those that employ generative models to obtain segmentation masks and those that employ purely discriminative methods such as contrastive learning (Ji et al., 2019a Ouali et al. 2020). Here, we focus generative approaches.
Nearly all generative approaches are based on the idea of decomposing the generative process in layer-wise fashion; in general, the foreground and background of an image are generated sepa- rately and then combined to obtain final image. Specifically, ReDo (Chen et al., 2019) trains a generator to re-draw new objects top of old objects, and enforces realism through adversarial training. (Bielski & Favaro 2019) generates a background, a foreground, and a foreground mask separately and composite them together; they prevent degenerate outputs (i.e. the foreground and background being the same) by randomly shifting the foreground relative to the background. Copy- Paste GAN (Arandjelović & Zisserman 2019) receives two images as input and copies parts of one image onto the other. OneGAN (Benny & Wolf, 2020) learns to simultaneously generate, cluster, and segment images with a combination of GANs, VAEs, and additional encoders. Equivariant Lay- ered GAN (Yang et al. 2021) first trains new layerwise GAN and then trains a segmentation net- work on synthetic data. Labels4Free (Abdal al. 202T) proposes new layerwise network inspired by StyleGAN for foreground bwwkouundbbakgooundd segmenation. Common difficulties with these layer- wise approaches above include the challenges of training new GANs and scaling beyond simple datasets (e.g. CUB, Flowers).
One work along different lines is by Voynov et al. (2021). which uses a pretrained BigBiGAN (Don- ahue & Simonyan, 2019) generator rather than proposing new layer-wise GAN. Voynov et al. (2021) use the method from Voynov & Babenko (2020) to decompose the latent space into inter- pretable directions. A recent version of their method also picks out this direction without supervi- sion using the idea that a breeground-backgroon two affine operators acting on different sets of pixels.
separating direction should be decomposable into
Our approach is based on generative modeling, but it differs from other approaches in that we seek general-purpor purpose method to find foregroumd/backgroomm non-layer-wise GANs rather than encoding it explicitly or searching for it manually. This enables us to leverage any of the numerous existing generators that have already been pretrained on millions of high-resolution images, rather than developing a new GAN architecture for this specific task. Differently from layer-wise approaches, our approach does not require training new GANs and does not rely on the assumption that the foreground and background of an image are independent. Differently from /Vonnvv et al. (202T), we show through experiments that our method can be applied to wide range of different GANs and find that it delivers superior performance object segmentation and saliency detection benchmarks. Finally, unlike any of these previous works, we demonstrate that our gnreground-backgroung replacement for saliency networks for the task of learning dense semantic image representations. By clustering these representations, are able to extend our method from segmentation to semantic segmentation.
structure implicitly encoded in standard,
four out of five
segmentation network can be used as a drop-in
METHOD
Let x E 3 x be a (color) image. A generator (network) is a function G R - NxW that maps code variables 2 to images I G(z). Optionally, some generative models come with an encoder function E : R HXW + RD which computes an approximate inverse of the generator (i.e. (()())) r).
3

---

Published as a conference paper at ICLR 2022
Step Finding informative code variations
Step 2: Learning segmentation model
Pettrfuttan dieettun
2rededie
Senthezs
bottinn deecttio v
ounstrnncy
G
dedeection V.
L
Figure 2: (Typo fixed) Our unsupervised segmentation pipeline. First (left), direction is identified in the latent space of a deep generative model (G) that separates the foreground and background of generated images by changing their relative brightness. Second (right), synthetic dataset is generated using this direction (or two of these directions) and is used train a separate segmentation network (S). This network can then be applied to unseen real-world data without further training.
challenge generating images that individual pixels exhibit complex correlations, caused by the fact that the images are obtained as the composition of number of different objects. For example, all pixels that belong to a dog have a similar color, characteristic of dog's instance. However, the correlation is much less strong between pixels that belong to different objects. This is because, while object in a scene are not entirely independent, their correlation is much weaker than within the structure of objects.
Intuitively, an image generator must learn to account for such correlations in order to generate realistically-lookingg images. In particular, we expect the generator capture the idea that pixels that belong to the same object have a related appearance, whereas the appearance of pixels that be- long to different objects or, as may be, a foreground object and its background, should be much more statistically independent.
Given a generator function G, it is then natural to ask whether such correlations can be extracted and used not just for the purpose of generating images, but also for analyzing them. In order to explore this idea, we consider perturbing the code via small increment eu E R., where e E RR and v E D is a unit vector. Because the dimension D of the embedding space is typically much smaller than the dimension 3HW of the generated images, codes provide highls-compressed views of the data (for example, D 120 for BigBiGAN (Donahue & Simonyan, (019) and the self-conditioned GAN), As such, most changes in the code are likely to affect most if not all pixels in the image. However, if the generator did in fact learn to compose objects, then one could hope too find specific variations u that only affect only portions of the image, corresponding to individual objects, and use the latter to highlight and segment them.
Empirically, we find that the situation is not as simple. Specifically, in the code that leave part of the pixels exactly constant while changing other pixels. However, we find that there are directions that affect foreground and background regions in a systematic and characteristic manner. Furthermore, the same works for all codes z, and are thus characteristic of a given generator network G.
not easy to find changes
show that these directions are 'universal", in the sense that
3.1 FINDING INFORMATIVE CODE VARIATIONS
Next, we introduce an automated criterion to select informative changes v in code space. To this end, we consider an image x G(z) generated from a random code z ~ 2, where 2 the code distribution characteristic of the generator (c.g. i..d. Gaussian). We then consider modified image x' = G(z + eu) and observe the change x + x'.
We compare the two images using two criteria. The first one preserves the structure of the image I. We capture the latter by imposing that x and x' generate approximately the same edges when fed to simple edge detector. The intuition is that wish to affect the appearance of objects without changing their shape. By preventing objects from 'moving around' the image or deforming, make it significantly easier extract image segmentation from the change x'. This loss
4

---

Published as a conference paper at ICLR 2022
Perturbation
Oepanel
Markc
(a) comparison generated images for different values (b) A comparison of perturbed images and their of the perturbation length using the BigBiGAN genera- masks for (foreground darker). vi, (foreground tor. A value of = 0.0 corresponds the original image, lighter), and the combination vs and vy. Using both with random Gaussian latent vector (0, 1).
directions yields visually superior segmentations.
takes the form:
(v(w) N
Σ
[S(G(z S(G
where ~ Z and S is the Sobel-Feldman operator:
[S(x)ly = Σ (g = Iea b + (g x.:))s
and g
2
11 -
This loss encourages x and x' to be similar. We thus also need a loss that encourages the direction to explore non-zero change of the image. We consider an image contrast variation and additionally exploit the photographer bias, that objects are often placed the middle of the image. This is captured by the loss:
(v(v) N
G( + te),
1=1 c=1
where r HR is 'radial' prior:
(V(1
H +
2
W 1+ 2
riy = -1-11aa aV
2 +
with normalization factor a - 1 that linearly interpolates from 1 in the center of the image to -1 at the boundary. This encourages finding direction that changes the brightness in the center of the image opposite to the border. In order learn v, the losses combined with weighting factor X.
((v) (C) + (v)
(I)
Given this fully-automatic procedure, the latent code direction v may be thought of as a function of the generator G and the weighting factor A.
3.1.1 COMBINING INFORMATIVE CODES
Optimizing Eq. (1) with À encourages the network produce shift that brightens the foreground and darkens the background of an image. However, there is no constraint that need be positive; by negating and optimizing second time, obtain another direction e that shifts the foreground dark and the background light.
Although using only direction suffices for method, find that by using both. As a result, for the remainder of the paper, let v represent the direction that shifts the foreground lighter, and va represent the direction that shifts the foreground darker.
improve performance
5

---

Published as a conference paper at ICLR 2022
3.2 LEARNING SEGMENTATION MODEL
Once the latent directions and have been found, the process of extracting segmentation mask straightforward: we label as foreground regions the pixels in which the imgee generated with the Woreground-lighte shifted latent code is lighter than the image generated with the foreground-darker shifted latent code. That is, for generated image x = G(2), we have:
((2) sign(G((z + evt) - G + eva))
(2)
Alternatively, if we use only a single direction by or va. M (z) is set to either:
G( + (vr) G(z) or G(z) G( (+ + eva)
(3)
Given the learned direction we use it to generate training set as follows:
D ((G())
2, I....)
This dataset may then be used train any dense segmentation network v (i.e., UNet (Ronneberger ett al. 2015)) in the standard fashion. That is, we minimize the pixel-wise binary cross-entropy loss between the segmentation output T (G())) Hx and the (synthesized) mask M()):
C(V(22)
(ogplsigm(MM.())))...CC x
Σ
HI
is a pixel index and sign is the sign function. Unlike pre- vious object segmentation methods, our method requires no additional losses or constraints to ensure the stability training. By sampling z, we can generate an 'infinite' dataset for learning the network V. Although we described the procedure above for unconditional GANs, our method applies just as well for weakly-supervised conditional GANs, where the generator G(z, y) also depends on class label; we simply sample a label uniformly at random for each generated image.
where p(m[s) (1-
3.2.1 REFINING THE GENERATED DATASET
An advantage of training with GAN-generated data is that the dataset size infinite, which means that one is free to curate one's dataset and discard uninformative training examples. In our case, we found that it was helpful refine the dataset by (1) discarding images with masks that were too large, (2) discarding images for which the latent code shift did not produce significant change brightness, and (3) removing small connected components from the mask. The exact details are given in the Supplementary Material.
EXPERIMENTS
In this section, we present an extensive set experiments demonstrating the method's performance, its wide applicability across image datasets, and its generalizability across GAN architectures.
4.1 EXPERIMENTAL SETUP
As our method generator-agnostic, apply our method to twelve generators, including three unconditional and nine conditional GANs. For the three unconditional GANs (BigBiGAN (Donahue & Simonyan 2019) SelfCondGAN (Liu et al. 2020), and UncondGAN (Liu et al. 2020)). our procedure is completely unsupervised. For conditional GANs, our method is still unsupervised but the GAN naturally relies on class supervision for training.
To demonstrate the efficacy of our method across resolutions and datasets, we implement both, GANs trained on ImageNet (Deng et al. 2009) at a resolution of 128px, and GANs trained on the smaller TinylmageNet dataset (100,000 images split into 200 classes) resolution of 64px. All ex- periments performed across all GANs utilize the same set of hyperparameters for both optimization and segmentation. This is key advantage of our method relative to other unsupervised/weakky supervised image segmentation methods (Chen et al.) (2119] Bielski Favaro 2019) Benny & Wolf, 2020 Arandjelović & Zisserman 2019), which are sensitive to dataset-specific hyperparameters.

---

Published as a conference paper at ICLR 2022
Table 1: (Updated for Voynov et al. (2021) Performance on three saliency detection benchmarks (DUTS, EC- SSD, DUT OUT-OMRON) and two object segmentation benchmarks (CUB, Flowers). ** initializes with pretrained supervised network. CRF post- processing. our implementation.
DUTS Acc
ECSSD IoU Fa
CUB
Flowers IoU maxFs
IoU
Ace
Acc
IoU
maxFa| Acc
Supervised Methods
Unsupervised Methods
Hou et al.
2019
00.924
0.729
0,880
Bicliki Fovtro 2019
Luo
2017
0.920
0.7360.934
0.891
ReDO
Chen al
2019
0.845 0.426
0,879 0.764
Zhang el
201
0.902
0.693 0.939
0.883
WNet
Xis Kalls
2017
0.248
Zhang
20170
0.868
0,660
0,852 UTSB Kanomli
2018
0.442
Wang el
201
0.915
0.672
0.908
0.826
IIC-seg
al
X019
0.365
c1
0.924
(0.440
0.759
OneCAN Benoy & Wolf 2020
0.555
Handcrafted Methods
Noynow
Vyyni of 7021
2021
0.931 0.693
0.930 0.683
0.794
0.807
0.765
0.777
0.540
0.529
0.760
0.672
RBD
7hu et al 4014
10.799
0.510
0.652
Ours (BigBiGAN ImageNet)
0.9221
0.664
0.783
0.779
0.541
0.723
DSR
2013
0.863
0.558
0.826
0.639
MC
Jing ai 2013
0.814
0.529
0.796
0.611
Komodakis
2015
0.773
007722
0.623
DUTOMRON
Deep Eissembles of Handerafted Methods
Acc
JoU
Fg
SBF
Zhung et al
2017
008665
0.583
0.787
Noynov et
2021
0.856
0.453
USD
Zhang
0.914
0.716
0.878
Voynow
2021
USPS
Nguyen
2019
0.938
0.736 0.937
0.874
Ours
0.883
0,509
0.583
Unsupervised Methods
Neynow et al. 2021
008788 0.498 (0.899 0.672
Noynow
0.881 0.508 0.600 0.60000.906 0.906 0.685 0.790
Ours
|0.893 0.528 0.614]0.9115 0.713 0.806
Table 2: A comparison of segmentation model performance across a wide range of generator architectures, using a foreground-lighter shift (vi). All hyperparameters are kept constant across generators. IN-128px refers ImageNet resolution 128px, and TinyIN-64px refers to TinylmageNet at resolution 64px.
CUB
Flowers
DUT-OMRON
DUTS
ECSSD
Dataset Res.
Ace
loU
Acc
loU
Acc
IoU
Ace
loU
Ave
IoU
ACGAN
Odena al
TinyIN 64px
0.682
0.265
0.572
0.266
0.642
0.190
0.647
0.652
0.276
BigGAN
Breek et
2019
Tmy EN 64px
0.853
0.257
0.723
0.284
0.844
0,213
0.842
0.224
0.811
0,332
GGAN
Lam Yo
Tiny 64px
0.818
0.697
0.782
0.783
0.766
0.316
SAGAN
Ziange
2719
TinyIN 64px
0.828
0.376
0.732
0.351
0.808
0.235
0.806
0.246
0.788
0.327
SNGAN
Miyato 45 al
2013
Tmy/N 64px
0.849
0.357
0.751
0.374
0.816
0.216
0.814
0.217
0.795
0.292
SAGAN
Zhang
3019
IN 128px
0.871
0.336
0.608
0.085
0.856
0,250
0.860
0.282
0.814
0.340
SNGAN
Mayalo st al
2013
IN 128px
0.881
0.378
0.703
0.304
0.860
0.305
0.854
0,300
0.837
0.432
ContraGAN Kang
128px
0.661
0,088
0.870
0,204
UnCondGAA
La at
X0X0
IN 128px
0.734
0.217
0.494
0.049
0.698
0.127
0.729
0.158
0.681
0.198
SelfCondGAN
In
128px
0.869
0.459
0.670
0.238
0.800
0.280
0.806
0.297
0.806
0.412
BigGAN Brock
IN 128px
0.886
0.367
0.731
0.318
0.883
0.316
0.876
0.303
0.848
0.424
RigBIGAN
Tomalun & Simongan 2019
IN 128px
0.912
0.601
0.773
0.479
0.878
0.451
0.890
0.486
0.905
0.663
4.2 EVALUATION DATA
We evaluate the performance of our model on three standard saliency detection benchmarks (DUT- Omrom (Yang et al., 2013), DUTS (Wang al. (017). ECSSD (Shi et al. 2016 and two standard object segmentation benchmarks (CUB (Welinder et al. 2010), and Flowers- 102 (Nilsback & Zisser- man 2009)). For the saliency datasets, evaluate using (pixel-wise) accuracy, mean intersection- over-union (IoU), and Fa-score with 3 = 0.3. For the object segmentation datasets, we evaluate using accuracy, IoU, and maxFs (the maximum F' score over range of 255 uniformly distributed binarization thresholds between 0 and 1).
4.3 RESULTS
Performance on Benchmarks. In Table we compare our method to other recent work. We em- phasize that method uses the same model for all datasets and has not seen any of the (training or evaluation) data for these datasets before. Our method delivers strong performance across datasets, approaching and even outperforming some superviseddhanddrratee saliency detection methods. In
7

---

Published as a conference paper at ICLR 2022
comparison to Voynov et al. (2021) ,, we perform better on four out of five benchmarks (all except CUB), even though we do not rely on humans to hand hand-pick pick latent directions. In comparison to
layerwise GANs (Chen al. 2019 Bielski & Favaro 2019; Benny & Wolf 2020 Arandjelović & Zisserman 2019), we perform similarly on CUB and Howeers112,, but we cannot compare our method with layerwise GANs on complex datasets (e.g. DUTS) because they do not produce any meaningful results. Due to the difficulty of training GANs, they are only ever trained on datasets consisting of images from a single domain with a single main subject, such as birds or flowers. By contrast, our ability to leverage pre-trained generators means that our method scales to complex and diverse datasets, such as those used for saliency detection.
Performance across Generators. We inves- tigate the generality of our method by perform- ing the same optimization and training pipeline with twelve different GANs. For each gener- ator, we optimize to obtain latent direction Ui train a segmentation model using this di- rection, and evaluate its performance across the
SNGAN
SNGAN (Tryth)
SAGAN
BIGGAR (16
D:90
ContraGAN (NN)
SAGAN (Tmn)N)
(Tiry1N
Avrrgg Acceracy
-G004 (T/5S1)))
DAM
ADSAN ADGAN (Tiny-00
five datasets above. The same hyperparame- ters are kept constant for all GANs, including = 5.0 during the optimization phase.
0:55
300
FID Score
Results are shown in Table 2 BigBiGAN per- forms best, but all networks, even those using relatively weak TinylmageNet-rrineed GANS (e.g., GGAN (Lim & Yel (0177), deliver rea- sonable segmentation performance. This high- lights the benefits of our aully-automatic seg- mentation pipeline; our method performs well across a wide range of generators trained on different datasets at different resolutions.
Figure 4: A plot of Frechet Inception Distance (FID) versus average segmentation accuracy across all five evaluation datasets (CUB, Flowers, DUT-OMMON, DUTS, ECSSD) for nine different GAN architectures. Lower FIDs are better (note that the x-axis is reversed). Lower FID scores correlate with improved final seg- mentation accuracy.
Naturally, the quality of final segmentation network produced by our method related to the quality of the underlying generator. Figure 4 plots the Frechet Inception Distance (FID) score of nine conditional GANs versus the average accuracy of the corresponding segmentation networks produced by our method. Lower FID scores, which correspond to better GANs, correlate with improved accuracy. This correlation suggests that as better GANs architectures are developed, our method will continue to produce better unsupervised segmentation networks.
Comparing latent directions. We compare the performance of a segmentation masks using the two latent directions Uy and vy together (Eq. (2. each of them individually (Eq. (3) visually in Fig. Bal In Table we quantitatively compare the results of these three methods along with a fourth method in which we ensemble the final segmentation networks produced by us and v individually. The foreground-lighte (vy) and foreground-darker (v:) directions yield similar results when used individually. The combination (u) and vg) provides superior results, on par with the ensemble. Unlike the ensemble, which requires training two networks, the combination of eg and vy adds minimal overhead compared to training with one direction.
Ablation:
Ablation: Varying X, the Central Prior, Random Initializations, Amount of Generated Data. The two hyperparameters in the optimization stage of our method are X, which controls the trade-off between brightness and consistency, and which controls the magnitude of the perturbation. We find that the process is only modestly sensitive to changes in these hyperparameters. We also find that using the central prior compared to spatially-agnostec loss term is better, but only moderately. Detailed numerical results, including the ablation on the central prior, are given in [Section A.3] In Appendix B.3 we also investigate how segmentation performance varies with the number of generated images, finding a clear log-linear relationship.
Qualitative Results. By inspection, we find that our optimization procedure is able to edit images in such a manner that the foreground becomes lighter and the background becomes darker. In nu- merous cases, the network appears convert the scene from daytime nighttime. Furthermore, better GANs generally produce qualitativley better segmentations. Please refer to Appppeixx A and Section A.1.3 for illustrations.
8

---

Published as a conference paper at ICLR 2022
Table 3: A comparison of semantic segmentation performance on Pascal- VOC obtained from K-means clus- tering of pixelwise semantic features. The mloU computed over the 20 classes by performing Hungarian matching between the clusters obtained from K-means and the ground truth. All networks backbone. We compare with numerous baselines, including using self-supervised features directly (i.c. MoCo, SwaV) and IIC. Compared to Van Gansbeke et al. (202T), we achieve competitive performance, but our pipeline is entirely unsupervised, whereas theirs uses saliency network which was initialized with supervised network pretrained for semantic segmentation SitySc
ResNet
Method
Saliency Network
Saliency Net. PT
Sem. Seg. PT
mloU
Colorization
Proxy task Clustering
Colorization
4,9
IIC
IIC
9.8
MoCo
Image Contrast
Moco
4.3
Swav
Image Contrast
Swav
4.4
ImageNet Sup.
Image Contrast
Sup. ImageNet
4.4
MaskContrast
Pixel Contrast
DeepUSPS + BAS-Net Cityscapes (Sup.)
MoCo
35.0
MaskContrast
Pixel Contrast
DeepUSPS BAS--Ne Net Cityscapes DUTS (Sup.)
MoCo
38.9
Ours (BigBiGAN)
Pixel Contrast
Our method
Our method (Unsup.)
MoCo
.3.55
4.4 EXTENSION To SEMANTIC SEGMENTATION
Finally, we demonstrate that our network may be extended from binary bnegground-bcckrronne segmentation semantic segmentation, the task of assigning each pixel in image into one of K semantic categories. Due to the challenging nature of this task, it has not been attempted by any previous works in the GAN-based segmentation space.
We extend our method by following the dense contrastive learning approach proposed by Van Gans- beke et al. (2021). In this approach, binary masks are extracted from set of images using segmentation model, sometimes called a "mid-level visual prior." Then, network trained to generate pixel-wise features using mask-based contrastive loss: features corresponding to pixels in the foreground of the image are pulled toward the features of other pix- els in the mask and pushed away from the features of background pixels. After training, semantic segmentations may be extracted by clustering these pixel-wise features across entire dataset.
Importantly, Van Gansbeke et al. (202T) uses saliency detection networks to generate their object masks. Although these saliency detectors are sometimes called "unsupervised." they are actually initialized using pretrained semantic segmentation networks]] We propose to use our object seg- mentation network as a drop-in replacement for these saliency networks, making the entire process entirely unsupervised. We leave the rest of their method (i.e. the dense contrastive learning and the evaluation procedure) unchanged.
We perform experiments on the PASCAL VOC dataset, which contains 20 semantic classes. Exper- imental details are included in Seccion A.1 K-Means clustering results shown in Table 3) We compare to a range of baselines, along with two models from Van Gansbeke et al. (2021) us- ing different levels of supervised pretraining. Our network is competitive with Van Gansbeke et al. 202II) despite being entirely unsupervised. Furthermore, in Table 4 in Seection 1.1.3 we show that this procedure works for wide range of GANs, demonstrating the generalizability of our approach.
5 CONCLUSIONS
We find that extracting salient object segmentation from the latent space of GAN is not only possible without supervision but also leads to ofate-of-the-art unsupervised segmentation perfor- mance several benchmark datasets. In contrast to existing methods that have been engineered specifically for this task, we extract segmentations from a network trained for a very different pur- pose generating images. Surprisingly, we able to generalize benchmarks without directly training on any real images, and even extend our results to semantic segmentation. Our findings directly prompt future research questions about what other concepts of the physical world can be automatically extracted from generative models, and to what extent we can use such extracted concepts to replace human supervision in other computer vision tasks.
wide range of segmentation
In the case of Van Gansheke et al. 2021). they use DeepUSPS al. 2019), which is initialized using supervised network pretrained for semantic segmentation Cityscapes.

---

Published as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
We aim to ensure that our experiments are entirely and easily reproducible. We upload code to the Supplementary Material fully reproduce all experiments. This code contains README file with a detailed description of the code structure, which should help enable others to reproduce and later extend upon our work. We also take care describe all hyperparameters and implementation details in the Appendix. Our results do not require extremely large amounts of compute; they can be reproduced with a single GPU by researchers with computational constraints.
7 ETHICS STATEMENT
is important to discuss the potential ethical issues involved with training large-scale generative models and segmentation networks along the lines proposed by our paper.
First of all, the task of segmentation is predicated upon classifying objects into predetermined (either binary or semantic) categories; the definition of these categories, especially in the case of semantic segmentation, may introduce biases into the task itself. Second, when training models on large- scale datasets, it is essential consider the biases and data privacy issues introduced in the data collection process. For example, the PASCAL-VOC dataset, which we use for the task of semantic segmentation, composed of images scraped from Flickr. As result, it is composed primarily of photographs from the United States and Europe, and the "person" class contains primarily images of white individuals. Additionally, it is not clear whether the individuals in these photographs consent to being used to train image segmentation models, bringing up the issue of data privacy.
From an ethical perspective, our method slightly different from standard segmentation models because it is trained solely on GAN-generated images; this is not to say that it is ethically better or worse, but that it involves different ethical considerations. On the one hand, this might alleviate some data privacy concerns, training data is generated by a GAN, one has to examine the data and methodology originally used to pretrain the GAN; any biases present in this data will likely be reproduced amplified by the GAN For example, if one uses a GAN trained on ImageNet to perform object segmentation, it may perform better on white individuals than individuals of other races due the disproportionate percentage of white individuals in the training data (Steed & Caliskan 202T). Investigating biases introduced by GANs remains an active area of research in the machine learning community (Jain) et al. 2020 Tan et al. 2020), and these ethical discussions extend to our GAN-based segmentation method.
the segmentation training data is synthetic. However, since this
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Labels4free: Unsupervised segmenta-
tion using stylegan. 2021.
Relja Arandjclović and Andrew Zisserman. Object discovery with a copy-pasting gan. arXiv preprint arxiv: /3369 2019.
David Bau, Yan Zhu, Jonas Wulff, William S. Peebles, Bolei Zhou, Hendrik Strobelt, and An- tonio Torralba. Seeing what GAN cannot generate. In Proc. ICCV, 2019.
Yaniv Benny and Lior Wolf. Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-grained clustering. Lecture Notes in Computer Science, pp. 514-530, 2020. ISSN 1611-3349. doi:
Adam Bielski and Paolo Favaro. Emergence of Object Segmentation in Perturbed Generative Mod- els. Proc. NewrIPS, volume 32, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In Proc. ICLR, 2019.
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised Object Segmentation by Re- drawing. In Proc. NeurIPS, volume 32, 2019.
10

---

Published as a conference paper at ICLR 2022
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Processing Systems (NIPS), 2016.
Deng. W. Dong, R. Socher, L. Li. Kai Li, and Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In Proc. CVPR, pp. 248 255, 2009. doi: 10.1 1110///VV..00095206666
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Proc. ICLR, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. volume 27, 2014.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv.es, abs/1911.05722. 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. Proc. ICLR, 2017.
Judy Hoffman, Erie Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In Proc. ICMI., 2018.
Q. Hou, M. Cheng, X. Hu, A. Borji, 7. Tu, and P. H. S. Torr. Deeply supervised salient object detection with short connections. PAMI, pp. 815-828, 2019. doi: 10.1109/TPAM.2211. 2815688.
Qiyang Hu, Attila Szabó, Tiziano Portenier, Paolo Favaro, and Matthias Zwicker. Disentangling factors of variation by mixing them. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1399-3407, 2018.
Niharika Jain, Alberto Olmo, Sailik Sengupta, Lydia Manikonda, and Subbarao Kambhampati. Im- perfect imaganation: Implications gans exacerbating biases on facial data augmentation and snapchat selfie lenses. arXiv preprint arXiv:2001.0955 2020.
Xu Ji, Joao F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised im- age classification and segmentation. In Proceedings of the International Conference on Computer Vision (ICCV). 2019b.
B. Jiang, L. Zhang, H. Lu, C. Yang, and M. Yang. Saliency detection via absorbing markov chain. In Proc. ICCV, pp. 1665-1672, 2013. doi: 10.1 1091/CCC.2211332200
A. Kanezaki. Unsupervised Image Segmentation by Backpropagation. In Proc. ICASSP, pp. 1543- 1547, 2018. doi: 10.1 109/ 3.66555....
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In H. Larochelle, M. Ranzato, Hadsell, M. Balcan, and Lin (eds.), Proc. NeurlPS, vol- ume 33, pp. 21357-21369. Curran Associates, Inc., 2020.
Tero Karras, Samuli Laine, and Timo Aila. A Style- Based Generator Architecture for Generative Adversarial Networks. In Proc. CVPR, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2. 2014.
X. Li, H. Lu, L. Zhang, X. Ruan, and M. Yang. Saliency detection via dense and sparse reconstruc- tion. In Proc. ICCV, pp. 2976-2988 2013. doi: 10.1 109/CC022013.370.

---

Published as a conference paper at ICLR 2022
X. Li, L. Zhao, L. Wei, M. Yang, F. Wu, Y. Zhuang. H. Ling, and J. Wang. Deepsaliency: Multi-task deep neural network model for salient object detection. In JEEE Trans. on Image Processing, pp. 3919-3930, 2016. doi: 10.1 0..119971.22011.25579000
Jae Hyun Lim and Jong Chul Ye. Geometric gan. In arXiv.cs, 2017.
Steven Liu, Tongzhou Wang, David Bau, Jun- Yan Zhu, and Antonio Torralba. Diverse image gen- eration via self-conditioned gans. In Proc. CVPR, 2020.
Sebastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and Nate Kushman. Inverse graphics gan: Learn- ing to generate 3d shapes from unstructured 2d data. arXiv preprint arXiv:2002. /2674, 2020.
Zhiming Luo, Akshaya Mishra, Andrew Achkar, Justin Eichel, Shaozi Li, and Pierre-Mare Jodoin. Non-local deep features for salient object detection. In Proc. CVPR, July 2017.
Michael Mathicu, Junbo Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann Le Cun. Disentan- gling factors of variation in deep representations using adversarial training. Advances in Neural Information Processing Systems, pp. 5047-5055, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proc. ICLR, 2018
Duc Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. DeepUSPS: Deep Robust Unsupervised Saliency Prediction With Superviseri Proc. NeurIPS, 2019.
M.-E. Nilsback. An Automatic Visual Flora - Segmentation and Classification of Flowers Images. PhD thesis, University of Oxford, 2009.
Maria-Elena Nilsback and Andrew Zisserman. Delving deeper into the whorl of flower segmenta- tion. Image and Vision Computing, 2009.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with aux- iliary classifier GANs. In Doina Precup and Yee Whye Teh (eds.), Proc. ICMI, volume 70 of Proceedings of Machine Learning Research, pp. 2642-2651, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive Unsupervised Image Segmenta- tion. In Proc. ECCV, 2020.
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. Proc. ICLR, 2021.
William Peebles, John Peebles, Jun-Yan Zhu, Alexei Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. arXiv preprint arXiv:2008. 10599, 2020.
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Do generative models know disentan- glement? contrastive learning is all you need. arXiv preprint 102. 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Proc. MICCAI, 2015.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. arXiv preprint 2xXiv:2007.06600 2020.
Shi, Q. Yan, L. Xu, and Jia. Hierarchical image saliency detection on extended essd. In PAMI, 2016.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2107-2116, 2017.
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training con- tain human like biases. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 701-713, 2021.
12

---

Published as a conference paper at ICLR 2022
Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the faimess of deep generative models without retraining. arXiv preprint arXiv:2012.04842, 2020.
Marco Toldo, Andrea Maracani, Umberto Michieli, and Pietro Zanuttigh. Unsupervised domain adaptation in semantic segmentation: a review. arXiv.es, 2020.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming- Siugn Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Pruc. CVPR, June 2018.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsuper- vised semantic segmentation by contrasting object mask proposals. In International Conference on Computer Vision, 2021.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In Proc. ICML, 2020.
Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large-scale generative models. In International Conference on Machine Learning, pp. 10596- 10606. PMLR, 2021.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Perez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In Proc. CVPR, June 2019.
Lijun Wang, Huchuan Lu, Yifan Wang. Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning detect salient objects with image-level supervision. In Proc. CVPR, 2017.
Wang, A. Borji, L. Zhang, Zhang, and H. Lu. A stagewise refinement model for detecting salient objects in images. In Proc. ICCV, pp. 4039 4039-4048, 2017. doi: 10.1 N0.1109//CCV2221...333
Welinder, S. Branson, Mita, C. Wah, F. Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Xide Xia and Brian Kulis. W-net: deep model for fully unsupervised image segmentation. In arXiv.es, 2017.
Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph-based manifold ranking. In Proc. CVPR, pp. 116663173. IEEE, 2013.
Yu Yang, Hakan Bilen, Qiran Zou, Wing Yin Cheung, and Xiangyang Ji. Unsupervised foreground- background segmentation with equivariant layered gans. arXiv preprint arXiv:2104.0044 2021.
Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang, Mingyang Qian, and Yizhou Yu. Multi-Source Weak Supervision for Saliency Detection. In Proc. CVPR, pp. 6074-6083, 2019.
Dingwen Zhang, Junwei Han, and Yu Zhang. Supervision by fusion: Towards unsupervised learning of deep salient object detector. In Proc. ICCV, Oct 2017a.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. Proc. ICML, pp. 7354-7363, 2019.
Jing Zhang, Tong Zhang, Yuchao Dai, Mehrtash Harandi, and Richard Hartley. Deep Unsupervised Saliency Detection: A Multiple Noisy Labeling Perspective. Proc. CVPR, 2018
Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan. Amulet: Aggregating multi-level convolutional features for salient object detection. In Proc. ICCV, Oct 2017b.
Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Baocai Yin. Learning uncertain convolutional features for accurate saliency detection. Proc. ICCV, Oct 2017c
W. Zhu, S. Liang, Y. Wei, and Sun. Saliency optimization from robust background detection. In Proc. CVPR, pp. 2811- 2014. doi: 10.1 (0.1109//VPR.2214.336
13

---

Published as a conference paper at ICLR 2022
W. Zou and N. Komodakis. Harf: fierarchy-associiae rich features for salient object detection. In Proc. ICCV, pp. 4064414, 414, 2015. doi: 10.1109/1CV220155544.
Yang Zou, Zhiding Yu, V. Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proc. ECCV, September 2018.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 OPTIMIZATION
First, we optimize for the latent code vectors vu and vt. We generate latent codes N 1) and optimize the vector ei (or va) by gradient descent with the Adam (Kingma & Ba 2014) optimizer and learning rate 0.05. We use for the light direction 2y and x= for the dark direction Ub- We perform 1000 optimization steps, by which point (or Ud) has converged.
A.1.2 OBJECT SEGMENTATION
(O, 1). produce the mask m (2), and refine the mask as described in the main paper. We use simple UNet (Ronneherger et al. 2015)) with bilinear down/up- sampling as our segmentation network. Following (Voynov et al. (021). we train for 12000 steps using Adam with learning rate 10-3 and batch size 95, decaying the learning rate by a factor of 0.2 at iteration 8000.
To generate data, we sample
A.1.3 SEMANTIC SEGMENTATION
We follow the procedure from Van Gansbeke al. 2021). but binary masks produced by object segmentation model as drop-in replacement for their saliency detection masks. Under this procedure, we begin by applying split of PASCAL-VOC (10582 images). We then train a dilated ResNet-50 model with DeepLab- v3 head, initialized with pretrained MoCo ResNet-50. As in (Van Gansbeke et al., 202T), this network has a main head and an auxiliary head: the main head predicts dense pixel-wise features of dimension 32, and the auxiliary head predicts the object segmentation mask for the purpose of regularization. We train the output of the main head using a pixel-wise version of the InfoNCE loss. Specifically, for feature 2; corresponding pixel in the foreground mask M,, we have:
segmentation network to extract masks from the tr naug
exp(zi 24/T)
L:
log
exp(zi x-(T)
where 24 is the average embedding of other pixels in the mask M,, and is the average embeding of pixels in the background:
JM
Zis
N
M igM,
IEM:
where N is the number of pixels in the image.
We use the same hyperparameters as (Van Gansbeke et al. 2021) for the contrastive learning pro- cedure. We train for 40 epochs using random crops of size 224, learning rate 0.01, and output dimension 32. After training, we evaluate with K-Means clustering with 21 clusters (20 seman- tic categories and background category). These clusters computed globally over the entire validation set and matched to the ground truth labels using the Hungarian algorithm.
We repeat this same procedure identically for large range of GANs. Figure 9 compares the result of final segmentation networks for different GANs: all images produce sensible segmentations, with better GANs generally producing better segmentations.
A.2 DATASETS
Below, we give statistics of our evaluation datasets. Note that these are only used for evaluation, as only GAN-generated images are seen during training:
15

---

Published as a conference paper at ICLR 2022
GAN
Dataset
Resolution
mloU
BigBiGAN
ImageNet
128px
0.345
SAGAN
ImageNett
128px
0.259
SelfCondGAN
ImageNet
128px
0.228
UncondGAN
ImageNet
128px
0.157
ContraGAN
ImageNet
128px
0.153
BigGAN
Tiny- Iiny-ImageNet
64px
0.124
SNGAN
64px
0.095
Table 4: Semantic segmentation performance on Pascal- VOC obtained from K-means clustering of pixelwise semantic features. mIoU is computed over the 20 classes by performing Hungarian matching between the clusters obtained from K-means the ground truth. We see that, as with object segmentation, better GANs (i.e. BigBiGAN, SAGAN) yield better downstream semantic segmentation performance.
CUB
Flowers
DUT-OMRON
DUTS
ECSSD
Acc
IoU
Acc
IoU
Acc
IoU
Acc
loU
Acc
IoU
or only
0,912
0.601
0.773
0.479
0.878
0.451
0.890
0.486
0.905
0.663
vd only
0.912
0.631
0.806
0.572
0.842
0.442
0.864
0.478
0.899
0.672
tt and
0.921
0.664
0.796
0,541
0.883
0.509
0.893
0,528
0.915
0.713
Ensemble
0.921
0.650
0.799
0.544
0.881
0.492
0.894
0.524
0.917
0.713
Table 6: comparison of segmentation performance when different directions in the latent space used to construct the training segmentation masks.
Dataset
Num. Images
Type
Crop
CUB
1000
Object seg.
Flowers
1020
Object seg.
OMRON
5168
Saliency det.
DUTS
5019
Saliency det.
X
ECSSD
1000
Saliency det.
X
Table 5: Evaluation dataset statisties
A.3 ADDITIONAL ABLATIONS
Comparing latent directions. In addition comparing the networks resulting from and Ub. we also compare the actual latent directions e and ug. Due to the nonlinearity of the generator function, the optimal unit directions v and Va are not necessarily negations of one an- other; indeed, we found in practice that they were close to but not exactly antiparallel. Table in Section A.3 gives exact numbers for variety of different generator architectures.
Ablation:
Ablation over X. The most notable hyperparameter in the optimization stage of our method is X, which controls the trade-off between brightness and consistency. Table 7 compares segmentation results for BigBiGAN with X 10,5, 2.5, and 1.25, with 5 performing best.
Ablation over 6. Table we show ablation results for changing during the optimization process. Note that since the GAN used in this set of experiments (BigBiGAN) has 120-dimensional latent space, the distribution the norm of the (0, 1) latent vectors used to train the GAN is concentrated around (approximately) 11. That is to say, shift of magnitude space very large.
6 in the latent
Ablation: Random initialization. Given the importance of the latent direction optimization pro- cedure in our segmentation pipeline, it is natural to ask whether different initializations yield mate- rially different directions in the latent space. To answer this question, optimized ten vectors
16

---

Published as a conference paper at ICLR 2022
CUB
Flowers
DUT-OMRON
DUTS
ECSSD
Acc
IoU
Acc
loU
Acc
IoU
Acc
IoU
Acc
loU
A= 10
0.911
0.631
0.794
0.550
0.849
0.455
0.874
0.498
0.899
0.677
À == 5
0.919
0.658
0.782
0.506
0.880
0.498
0.891
0.524
0.912
0.703
À = 2.5
0.818
0.418
0.728
0.456
0.762
0.311
0.765
0.311
0.792
0,467
À 1.25
0.791
0.385
0.713
0.449
0.740
0.296
0.743
0.296
0.771
0.446
Table A comparison of segmentation performance for a BigBiGAN model when different values of À are used to find the latent vectors i and in the optimization stage. Higher values of X yield latent directions v that produce shifted images with greater variance in brightness between the center and outside pixels. Conversely, lower values of À yield latent directions that produce shifted images that align better to the original images.
CUB
Flowers
DUT-OMRON
DUTS
ECSSD
Acc
IoU
Acc
ToU
Acc
IoU
Acc
IoU
Acc
IoU
0,911
0.600
0.744
0.600
0.867
0.454
0.880
0.479
0.897
0.650
0.912
0.601
0.773
0.479
0.878
0.451
0.890
0.486
0.905
0.663
0.843
0.435
0.617
0.435
0.763
0.290
0.775
0.297
0.779
0.419
0.761
0.347
0.602
0.347
0.714
0.236
0.709
0.238
0.724
0.349
Table 8: A comparison of segmentation performance for a BigBiGAN-based model when differ- ent values of are used to find the latent vector i/ in the optimization stage. Higher values of correspond to a greater-magnituee shift in the latent space during optimization.
BigBiGAN from different initializations and measured their pairwise cosine similarities. We found that the cosine similarities had mean 0.9866 with standard deviation 0.0044, which extremely small relative to the values in Table 9] Thus, the random initialization of the optimization procedure does not have large impact on the final direction.
On the necessity of the center prior. Here, conduct small experiment to inves- tigate the necessity of the central prior in the optimization step our segmentation pipeline. We replace it with image to have high variance in brightness. We find that this spatially-agnostic loss still yields good results, although not quite good the spatial prior loss. Numerical results shown in Table 10.
Ablation:
loss term with no central prior: it simply encourages the shifted
A.4 ADDITIONAL RELATED WORK
Saliency Detection. Object segmentation is closely related to saliency detection, the problem of finding significant (salient) objects in an image. Although most saliency detection models are trained with pixel-level supervision (or human eye movement data), the past few years have wttpe/iiiedwwwall--sspeeereeeeee
saliency detection (Zhang et al., 2018 Nguyen et al.) 2019 Zeng al. 2019). These methods work by ensembling strong hand-crafted priors and dis- tilling them into a deep network. In practice, they also initialize their networks with pretrained (supervised) image classifiers or semantic segmentation networks.
search into
Learning from Synthetic Data. Finally, our method can be viewed from the perspective of learn- ing from synthetic data. Motivated by the costly and time-consuming nature data labeling. there has been a plethora of recent work on synthesizing, training with, and adapting from synthetic datasets. For example, one widely-studied line of research (Hoffman al. 2018 Tsai et al. 2018 Zou et al., 2018 Vu et al. [2119] Toldo et al. 2020) tackles the task of semantic segmentation by training on data generated from video games (c.g. GTA5). With regard to adversarially-genenas
training data specifically (Shrivastava et al. 2017) of synthetic images while preserving label information.
GAN-like network to enhance the realism
Although train segmentation network using generated images only. show in the exper- iments below that it generalizes to real-world images without the need for additional adaptation. Such additional adaptation could be an interesting avenue for future research.
17

---

Published as a conference paper at ICLR 2022
GAN
1/g 4b
BigBiGAN
-0,4376
SelfCondGAN
-0.7854
UncondGAN
-0.3522
ContraGAN
-0.3297
SAGAN
-0.4648
Table 9: The dot product the optimized foreground- -ighhtr (ut) and foreground- latent di- rections (1z) for different generators, all of which have 120-dimensional latent space. Across the board, the directions are almost but not exactly antiparallel (random vectors in this space have an expected dot product of with variance 120
CUB
Flowers DUT-O.
DUTS
ECSSD
Acc.
0.885
0.788
0.825
0.849
0.876
IoU
0.535
0.553
0.379
0.416
0.609
maxFs
0.672
0.689
0.473
0.526
0.728
Table 10: Ablation results for the spatially-agnostee spatial mask
A.5 ADDITIONAL EXAMPLES
A.5.1 EXAMPLES ACROSS DATASETS
In Figure 6 Figure aadd Figure 8 we show the results of applying our final segmentation network random images from each the five datasets on which we evaluated.
A.5.2 EXAMPLES ACROSS GENERATORS
In Figure 9 we show examples of pairs of generated images and the corresponding extracted samples for a range of different GANs.
A.5.3 SEMANTIC SEGMENTATION EXAMPLES
In Figure 10 we provide randomly selected examples of our semantic segmentations after K-Means clustering the PASCAL-VOC dataset. We see that masks of similar visual quality to those from MaskContrast (Van Gansbeke et al. 2021 despite the fact that our method entirely unsupervised.
18

---

Published as a conference paper at ICLR 2022
Perturbation Amount (#)
4.0
BigBiGAN
SelfCondGAN
ContraGAN
SAGAN
StyleGAN (AHFQ)
Figure 5: Examples of perturbed images generated by our method for five different generators (GANs). Note that the final generator, StyleGAN (Karras al. 2019). is only trained on close-up portraits of animals, and thus cannot be used for general-perpose image segmentation. Nonetheless, our method successfully identifies the foreground and background of the generated animal portraits.
19

---

Published as a conference paper at ICLR 2022
CUB
Flowers
Figure 6: Examples of the final segmentation network across evaluation datasets. From left right: original image, ground truth, prediction.
20

---

Published as a conference paper at ICLR 2022
DUT-OMRON
DUTS
Crufts
Figure 7: Examples of the final segmentation network across evaluation datasets. From left right: original image, ground truth, prediction.
21

---

Published as a conference paper at ICLR 2022
ECSSD
Figure 8: Examples of the final segmentation network across evaluation datasets. From left right: original image, ground truth, prediction.
22

---

Published as a conference paper at ICLR 2022
RigHiGAN (ImageNet)
SNGAN (ImageNet)
BigGAN (TinglmageNet)
SAGAN (ImageNet)
BigGAN (ImageNet)
SAGAN (TinylmageNet)
SelfCondGAN (ImageNet)
ContraGAN (ImageNet)
SNGAN (TinytmageNet)
Figure 9: A comparison of perturbed images and their corresponding masks for many different generators.
23

---

Published as a conference paper at ICLR 2022
Imuge
Ground Truth
MaskContrast
Ours
Image
Ground Truth
MaskContrast
Ours
Figure 10: Examples of semantic segmentations after K KMMeans clustering on the PASCAL-VOC dataset. We see that our masks are of similar beke al. 202T), despite the fact that our method entirely unsupervised. Note that these images are randomly selected, not cherry-picked.
visual quality to those from MaskContrast (Van Gans-
24

---

Published as a conference paper at ICLR 2022
B FURTHER EXPERIMENTS
Based on the helpful comments of our reviewers, we have conducted new experiments and created more visualizations to better understand how our method functions and performs.
B.1 ADDITIONAL EXPERIMENTS ON FLOWERS
In this section, we provide a comprehensive explanation and analysis of our performance on the Flowers dataset.
The Flowers dataset is different from the other datasets investigated in our paper (CUB, DUTS, DUT-OMRON, ECSSD, PASCAL VOC) in that the "ground-truth" masks for Flowers were obtained using an automated segmentation method rather than human annotations. As result, the ground truth are very unreliable. In fact, there are many instances in the Flowers dataset in which the ground truth mask is completely or nearly completely empty despite there clearly being flower in the image. This issue is also mentioned by Chen et al. (2019), who claim that their method "provide[s] better masks" than the ground-truth in cases of disagreement.
Below, we present an additional experiment to improve our performance on Flowers. We switch the underlying GAN in method from BigBiGAN (trained on ImageNet) to an unconditional StyleGAN2 trained on Flowers.
This experiment aims to show that method works well in-domain data (i.e using GAN trained on Flowers to segment Flowers) in addition to out-of-domain data (i.e. using a GAN trained on ImageNet segment Flowers).
Specifically, we found publicly available weights on GitHub for an (unconditional) StyleGAN2 model Karras et al. (2019) trained on the Flowers dataset at resolution of 256px, and we applied our method directly to this model. We use perturbation radius r 0.2 rather than r = 2.0 as in BigBiGAN because the latent space of StyleGAN2 has a different scale. Apart from this sin- gular parameter, however we apply our method without any hyperparameter tuning or additional modifications for the new StyleGAN2.
Results are shown in Table II We find that using StyleGAN2 dramatically improves the visual quality of our segmentations both quantitatively and qualitatively. Quantitatively, our performance under this setup is similar to other unsupervised methods which have been extensively finetuned for this task, such as Chen et al. (019).
The most severe failure cases involve empty ground- truth masks. Similarly to ReDo, we find that when our method and the ground truth disagree, our segmentations are often visually superior. At this level of performance, the high level of label noise means that exact mloU/accuracy numbers are not mcaningful. However, there is no doubt that switching generators to StyleGAN tremendously improves our flower segmentation performance.
We show examples of failure cases in Fig. 13
Additionally, we sought to identify the extent to which this improvement was due purely to the increased resolution (256px) opposed to the underlying GAN. We train segmentation model 2e8xx-oowssampled versions of the StyleGAN2 generated images and show the results in Table Higher resolution training improves performance modestly on the Flowers dataset, while the bulk of the improvement is due to the use of StyleGAN. This result supports our hypothesis that better GANs correspond to better unsupervised object segmentation performance under our model.
B.2 ADDITIONAL EXPERIMENTS ON FACES
In this section, we provide additional experiments on the domain of face images.
This section designed with comparison to Abdal et al. (202I) in mind. However, it is difficult to compare with Abdal et al. (2021) because they evaluate non-standard datasets, they do not release any information about their data splits, and they do not release code or pretrained models. CelebA-HQ-Me
In particular, they conduct evaluations from the dataset, but they do not specify which images these are. Nevertheless, we tried to recreate their setup as best possible establish comparison.
using 1000 randomly selected images
25

---

Published as a conference paper at ICLR 2022
We applied method to StyleGAN2 trained FFHQ. We use resolution 256px due time constraints (and 1024px would likely improve results).
We use perturbation radius 0.2, which is copied directly from the experiment in the section above (because both experiments use StyleGAN2). Again, with the exception of this single param- eter, did not change single other hyperparameter in our entire learning setup demonstrate the generalizability of our approach.
Results are shown in Table [2 Qualitatively, we find that model tends to segment the hair and face but not the clothes, which is an equally valid segmentation but does not align with the ground truth where the person's 's body is included in the foreground mask. This result could mean that StyleGAN2 internally models the clothing in a different manner from the face which would lead to this behavior in latent space.
B.3 DATA SIZE
In this section, we investigate the relationship between the amount of synthetic training data and segmentation performance. All experiments in the main paper use 1,000,000 generated examples.
We train models for the same number of total iterations using a varying number of synthetic training images from 100 to 1,000,000. Results are shown in Table [4] and Fig. [ We find consistently across all datasets that similar supervised learning- performance scales logarithmically with the dataset size. Differently from supervised learning, our labels are entirely free to generate, so we can scale our synthetic dataset as large as desired.
B.4 SEMI-SUPERVISED LEARNING
In this section, investigate whether our synthetic data can be used to augment existing supervised datasets in semi-supervised learning setup.
We consider subset of the CUB dataset with 1000 labeled examples along with synthetic data from our BigBiGAN SraageNet--retraineee model. Results are shown in Table B
First, we train supervised model on the 1000 images with ground truth segmentations as baseline. Similarly, we train a fully unsupervised model on our generated data alone. Now, in the semi- supervised setting, we have access to 1000 labeled images. When training on the combination of both datasets we can already see an improvement over both baselines in IoU (which is the harder and less saturated metric). With access some in-domain data, we can also take another step to reduce the domain gap between the generated and real images: we can filter our generated images by their distance to the supervised samples. Specifically, we select the 50000 nearest neighbors in our generated set (according to cosine similarity using supe-vupe ResNet-500He et al. (2019). and we refer to these as the "kNN generated images" in Table 13
We see that model trained on the combined ground truth and kNN samples surpasses both baselines and the supervised model. This result demonstrates that our pipeline can play a valuable role in semi-supervised object segmentation in addition to unsupervised object segmentation.
B.4.1 FAILURE CASES
In this section, we describe common failure cases of our models.
Fig. 12 Fig. 13 and Fig. 14 show examples of failure cases of our model.
One of the main failure cases of our method is to segment additional foreground objects beyond the "main object" the image. For example, on the CUB dataset (Fig. 12). we often segment objects such as branches and bird feeders alongside the bird in the image. Across all datasets, one of the other main sources of errors is the object boundary that is not as precise as the object. This could potentially be improved in the future through the use of higher-resolution GANs by post- processing (e.g. with a CRF).
The Flowers dataset is special case in which the largest source of error is label noise in the ground truth annotations. Occasionally, our method misses to segment a flower in the background or seg- ments too much of the stem.
26

---

Published as a conference paper at ICLR 2022
With regard to failure cases of mask generation, since there is ground-truth associated with these generated images, these failure cases were found by manually filtering approximately 500 generated images (Fig. [4). Common failure cases include segmenting too much or too little of the image and are often attributable to low-quality images generated by the GAN. Thus, improvements in the generation quality of GANs and the performance of our method are closely linked.
27

---

Published as a conference paper at ICLR 2022
Data
Acc
IoU
Chen cl 2019
Flowers
0.879
0.764
Ours (BigBiGAN)
ImageNet
0.796
0.541
Ours (StyleGAN)
Flowers
0.882
0.723
Table 11: Performance the Flowers dataset compared to prior methods. It is important em- phasize the fact that the "ground truth" of the Flowers dataset was generated using an automatic procedure and is extremely noisy. For example, significant fraction of the ground truth masks are entirely empry, and those that are not empty often do not properly reflect the content of the image. As a result, quantitative numbers on the Flowers dataset should be heavily discounted. We encourage the reader to Fig. 13 for visualizations of failure cases.
Acc
IoU
Labels4Free (StyleGAN UNet) Ours (StyleGAN UNet)
0.91
0.82
0.85
0.80
Table 12: Performance on 1000 random images from the CelebA-HQ-Mask dataset. Performance is exactly comparable because these models were evaluated using different random subsets.
Real Images
Gen. Images
Acc
IoU
Real images only
1000
0.924
0.586
All generated images only
0
1000000
0.906
0.616
kNN generated images only Combined: real images and all generated images Combined:
0
50000
0.908
0.588
1000
1000000
0.914
0.628
real images and kNN generated images
1000
50000
0.931
0.665
Table 13: Results of combining real images with generated images on the CUB dataset. We use subset of 1000 training images from CUB along with 1,000,000 generated images. Additionally, we evaluate using filtered subset of our generated images containing the 50,000 nearest neighbors to real images. The distance between images was computed using a self-supervised ResNet-50 He et al. (2019). We see that combining real data points with nearby generated data points gives the best results.
28

---

Published as a conference paper at ICLR 2022
Num. Images
CUB
Flowers
DUTS
OMRON
ECSSD
Acc
IoU
Acc
IoU
Acc
loU
Acc
IoU
Acc
IoU
1000000
0.923
0.653
0.798
0.540
0.890
0.524
0.875
0.492
0,914
0.709
300000
0.906
0.616
0.769
0.480
0,876
0.498
0.859
0.460
0,901
0.674
200000
0.903
0.606
0.784
0.508
0,868
0.473
0.852
0,440
0.898
0,659
100000
0.900
0.596
0.762
0.467
0.861
0.443
0.845
0.408
0.881
0.617
10000
0.874
0.486
0.740
0.412
0.849
0.374
0.835
0.345
0.848
0.505
1000
0.847
0.419
0.700
0.343
0.821
0.311
0.811
0.299
0.802
0.410
100
0.752
0.256
0.650
0.284
0.727
0.203
0.728
0.206
0.710
0.271
Table 14: Performance of method for different numbers of generated images. All models
trained for the same number of iterations (20000). For a visual representation of these numbers, see
Fig. 11
Resolution
Acc
Fi
IoU
128
0.878
0.795
0.715
256
0.882
0.798
0.723
Table 15: Performance of our method on the Flowers dataset using StyleGAN 2 model trained on Flowers for different input image resolutions. Using a higher resolution improves performance slightly.
0.60
0.55
Average IOU Across Five Datasets
0.50
0.45
0.40
0.35
0.30
0.25
102
10a
104
10a
10c
Number of Generated Images (log scale)
Figure 11: Here plot the average average mloU score of segmentation model we vary the number of synthetic training images and masks generated using our method. We see that model performance consistently improves with the number of generated training images, following log- linear scale.
29

---

Published as a conference paper at ICLR 2022
Image
Prediction
Ground Truth
Image
Prediction
Ground Truch
Examples of failure cases of our method on the test set of CUB. Differently from the ground truth, our method frequently segments foreground objects that are not birds, such as branches and bird feeders. This reflects the fact that model captures general foreground structure in images and has not been trained on CUB.
Figure 12:
30

---

Published as a conference paper at ICLR 2022
Image
Prediction
Ground Truth
Image
Prediction
Ground Truth
Figure 13: Examples of failure cases Flowers in which the ground- truth mask empty.
31

---

Published as a conference paper at ICLR 2022
Image
Mask
Image
Mask
Image
Mask
$0
Figure 14: Examples of failure cases for GAN-generated images and masks. As there is no ground- truth associated with these generated images, these failure cases were found by manually filtering approximately 500 generations.
32


Reviews:
[{"review_id": "jgPrI6FTy81", "reviewer": "Reviewer_zTVT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed to mine useful semantic concepts  from deep generative models which is an interesting and unexplored topic.\nAlthough the core idea of this work is to find plausible directions for latent space manupilation which is similar to [1].  The designed loss functions of edge preserving and image contrast enhancement are reasonable and achieve better image segmentation performance.\n\nHowever, I still have several concerns.\n1. The Sobel Filter is not perfect on extracting the foreground object contour as shown in Fig.2. Therefore, the optimized latent space manipulation direction may not be accurate.\n2. The motivation for the image contrast variation is not very plausible. The area covered by the brighreness mask is darked instead of the foreground object which means the mask generation (Eq. 2) not make sense. \n3. The authors should add more comparisons with similar approaches such as [2]. I suggest the authors follow the evaluation setting in [2]. The proposed method could synthesize face segmentation samples via StyleGAN2 pretrained on FFHQ. Then train a segmentation model and evaluate it on Celeba-HQ. \n4. Most experimental results are based on low resolution images. Could the authors provide more results on high resolutions such as 256x256 or 512x512. \n5. The latent manipulation may not be perfect. Please provide some failure cases for mask generation and further discussion.\n\n[1] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Big gans are watching you: Towards unsupervised object segmentation with off-the-shelf generative models. arXiv.cs, abs/2006.04988, 2020.\n[2] Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. 2021.", "summary_of_the_paper": "This paper presents a new unsupervised algorithm for image segmentation via deep generative models.\nSpecifically, the authors combined edge preserving and brightness enhancement loss functions to search the latent manipulation directions which only adjusting the foreground object brightness. As a result, the directions could be utilized to synthesize image segmentation training samples.\nExperimental results on saliency detection and image segmentation demonstrate the effecteness of the proposed method.   ", "summary_of_the_review": "Overall, the reasearch topic is interesting and the proposed method seems to be noval. The experimental evaluation is not very convicing. I may change the rating score If above issues are handled well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MSP0HLAGkL3", "reviewer": "Reviewer_XSNi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of the paper and the overall method is very interesting. The authors prove that by using their method, they can leverage and find foreground-background information in virtually any GAN architecture and use those cues to create a synthetic dataset that is then used for image segmentation. I think that this has a lot of applications and is an interesting area of research.\n\nThe paper is fairly well written and the overall message is well understood. I think that the authors demonstrate that their method can be applied to various architectures and that overall it has potential on identifying the foreground-background information.\n\nHowever, I find the rest of the experimental part relatively weak. More exactly, while it is interesting to find that the method works with 12 other GANs, I feel like that for the image segmentation part there are some missing key aspects. The first one, is the comparison with methods that do not use/are not generative. While I understand the motivation and clearly that the goal is to directly compare agains generative methods, I think that a comparison with all methods is needed. This will give a clear indication on how the generative methods compare against other methods. \n\nMoreover, I feel like a lot more details regarding what data they use for training the models is missing. In order to understand the power of this generated data, I feel like the authors should have answered to the following questions, but in the current form of the paper are missing: \n1) How many samples does the final dataset contain?\n2) How does the number of samples affect the performance of the final model?\n3) What happens if you combine data from multiple GANs?\n\nLastly, since the overall performance is relatively on par with other methods, while having some advantages in terms of supervision, I would have wished to see at least one experiment that augments existing datasets with synthetic data. In this form, while I like the idea, the numbers do not seem that convincing to me.", "summary_of_the_paper": "The paper tackles the problem of image segmentation. It focuses on finding foreground-background image separation cues from a pre-trained GAN and create a synthetic dataset. Once the dataset is created, it trains a segmentation models and test the approach on several benchmarks. The paper is centred around how to extract and detect this foreground-background separation in an unsupervised way. The method works with any GAN architecture and in the end achieves competitive results while being unsupervised.", "summary_of_the_review": "Overall I think that the paper is well written. My major concerns when it comes to the decision are related to the comparison with non generative methods and the lack of insights and information over the data used for training. Please see above for more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4lerEoB_-ux", "reviewer": "Reviewer_JFoE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The approach is well-described and the paper is generally easy to follow.\n- The literature review is comprehensive.\n- The approach seems to produce good qualitative results, and quantitative results are slightly better than competing unsupervised approaches.\n- Quantitative results are computed on a variety of tasks, not just instance segmentation.\n\nWeaknesses:\n- In my opinion, there is little novelty. Except for the latent direction discovery step, the overall framework is very similar to existing approaches.\n- The latent direction is found by exploiting the photographer's bias (i.e. objects usually appear at the center of the image). How does the method perform if this condition is not satisfied?\n- There is no discussion of failure cases (I think they would make the paper stronger).", "summary_of_the_paper": "The paper proposes an unsupervised approach for instance segmentation that leverages pretrained GANs. The approach is divided into two steps: first, the authors discover a latent direction in the generator that allows them to segment generated images. Then, the resulting image-label pairs are used to train a segmentation network. This network can finally be used to perform inference on unseen images. The approach is evaluated on a variety of GAN architectures. The main contribution claimed by the authors is that their method can be applied to off-the-shelf pretrained GANs, whereas competing approaches require architectural variations and re-training.", "summary_of_the_review": "Although the paper is well-written and its implementation seems to be technically sound, my main concern is the limited novelty of the proposed approach. There are already many existing approaches that leverage GANs in order to segment images, and while this approach presents some useful ideas, I think they are somewhat incremental. Overall, the paper seems to be below the bar, but I would also like to hear the author's opinion and the other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4M7UtJiBTxC", "reviewer": "Reviewer_YWmx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== PROS ===\n+ This paper tackles an interesting problem : that of forging an image segmentation model without supervision. It does so in an interesting way, by exploiting a latent structure in image generation models. While the motivation in intro and p4 is a bit handwavy, it seems to be verified by experiments. \n+ The proposed approach is very simple and can be adapted to any generative model. The experiments further show that the performance of the segmentation is correlated with the PID, a proxy measure for the quality of generated images.\n\n=== CONS ===\n- The performance on the Flowers dataset is a bit disappointing w.r.t. to other unsupervised methods. Could it be because of the domain on which the initial generative model was trained on? If the generative model was specialized in flower species, would the results change?", "summary_of_the_paper": "This paper describes a simple procedure that allows to train an unsupervised image segmentation model only using a generative model as input. By tricking the generator to predict proxy segmentation masks along with a generated image, the method allows to construct a possibly infinite dataset of images associated with a F/B mask. A segmentation model can then be trained on this dataset to predict foreground.\nThe proposed approach is evaluated across a wide range of small-scale segmentation datasets, and compared with various approaches to the problem (both supervised, handcrafted and unsupervised). An adaptation of this approach to semantic segmentation is proposed and evaluated on Pascal VOC.", "summary_of_the_review": "Overall, I think that this is an interesting paper, showing that generative models contain a lot of information about the structure of images. Being able to build a segmentation model that really works on real data is quite impressive. As stated by the authors, this indeed is an interesting application of generative models that is useful on actual CV tasks. Because of the surprising observation and decent experimental execution, I think this paper deserves to be presented at ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that this paper proposes a very interesting approach of finding useful information encoded inside a generative model. They show how foreground/background semantics learnt in a generative model are useful for tasks like segmentation.
This is a general approach that can be applied to other models in the future.
It is an accept.
Decision: Accept (Poster)