Year: 2021

Abstract:
Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs. In this paper, we develop an automatic procedure for finding directions that lead to foreground-background image separation, and we use these directions to train an image segmentation model without human supervision. Our method is generator-agnostic, producing strong segmentation results with a wide range of different GAN architectures. Furthermore, by leveraging GANs pretrained on large datasets such as ImageNet, we are able to segment images from a range of domains without further training or finetuning. Evaluating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data. Broadly, our results demonstrate that automatically extracting foregroundbackground structure from pretrained deep generative models can serve as a remarkably effective substitute for human supervision.

heading: 1 INTRODUCTION
text: Recent years have seen rapid progress in the field of deep generative modeling of images, driven by a proliferation of research into Generative Adversaial Networks (GANs) (Goodfellow et al., 2014). Nowadays, it is possible to generate high-resolution images of realistic objects and scenes (Karras et al., 2019; Brock et al., 2019). However, with the exception of generation for computer graphics, there has been limited research into how we might be able to leverage the representations learned by these powerful generative models to enhance other tasks, particularly those involving semantic understanding (e.g. classification or segmentation).
Generally, deep generative models learn to map latent codes to images, imposing simple statistical structures on the distribution of the latent codes, such as assuming an i.i.d. Gaussian distribution. Due to this structure, in some cases code dimensions acquire specific meanings which can be related to human-interpretable concepts (e.g., the rotation or size of an object); however, the code space in high-quality generators (e.g., BigGAN (Brock et al., 2019), BigBiGANs (Donahue & Simonyan, 2019), StyleGAN (Karras et al., 2019)) is usually not easily interpretable. Nonetheless, it is intuitive that an efficient generative process should account for the structure of natural images, including for example the fact that images often comprise distinct foreground and background regions.
In this paper, we validate this hypothesis by learning to separate foreground and background image regions from generator networks. Our approach starts from an arbitrary, off-the-shelf high-quality generator network trained on a large corpus of (unlabeled) images. While these generator networks are not explicitly trained for foreground/background segmentation, we show that such a separation emerges implicitly as a step to efficiently encode realistically-looking images. Specifically, we design a probing scheme that can extract such foreground/background information automatically, i.e. without manual supervision, from the generated images.
This scheme works as follows (cf. Fig. 1). We start from a random code in latent space and learn a fixed, global offset that results in a change in the generated images. The offset is learned to alter the appearance of foreground and background such that a mask can be extracted from the changes in image space.
The resulting masks provide segmentation maps for the generated images, but they cannot yet be used to segment images from the real world. Given a natural image, the obvious approach would
Under review as a conference paper at ICLR 2022 Top images were generated with seeds 5432 and 5431
G en
er at
ed im
ag es
an
d ex
tr ac
te d
m as ks Re al -w or ld im ag es a nd pr ed ic te d m as ks G en er at ed im
ag es
an
d ex
tr ac
te d
m as ks Re al -w or ld im ag es
an d
pr ed
ic te
d m
as ks
Figure 1: We automatically find a universal latent direction in a GAN that can separate the foreground from the background without any supervision. We can then generate unlimited samples with masks to train a segmentation network that achieves state-of-the-art unsupervised segmentation performance.
be to find the corresponding code in the latent space of the generator, and then obtain a mask with our method. Unfortunately, this inversion process is less than trivial. In fact, recent work provides strong evidence that the expressiveness of GANs is insufficient to encode arbitrary images (Bau et al., 2019), meaning that the inversion problem has no solution in general.
As we aim to build a general-purpose segmentation method, we take a different approach: we generate a labelled image dataset with foreground/background segmentations and use the generated dataset to train a standard segmentation network. With this, we show that our method can successfully learn accurate foreground-background segmentation networks with no manually provided labels at all. Differently from prior work in GAN-based image segmentation, we neither design a new GAN architecture specifically for the task of segmentation nor use manual supervision to extract segmentation information from an existing GAN. Thus, we can discover meaningful latent directions for any GAN with no need for model-specific manual intervention.
Extensive experiments on five segmentation datasets across twelve different GANs demonstrate the effectiveness and generalizability of our approach. Moreover, by constructing our image segmenters from generator networks trained on a generic large-scale datasets such as ImageNet, our method can learn to generically segment objects from a wide range of visual domains. Specifically, when we apply our generic segmenters to the CUB200 (Welinder et al., 2010) and Oxford Flowers (Nilsback, 2009) datasets, we attain very strong foreground-background results despite not training on this data. Similarly, when we apply our generic segmenters to three saliency detection benchmarks, our method approaches and sometimes even exceeds the performance of supervised and handcrafted saliency detection methods. An analysis of our results also shows that segmentation performance directly correlates with the quality of the underlying GAN, suggesting that foreground/background separation is an important concept in learning generative models.
Finally, we demonstrate that our method may be used as a drop-in replacement for saliency networks for the purpose of learning pixel-wise semantic image representations. These pixel-wise representations can then be clustered to obtain fully unsupervised semantic segmentations, extending our method beyond foreground-background segmentation. Thus, we not only demonstrate for the first time that it is possible to obtain semantic segmentations using GANs, but also that this information may be extracted from a wide range of generic GANs trained on general-purpose image datasets.

heading: 2 RELATED WORK
text: Below, we describe how our method relates to recent work in interpreting generative models and object segmentation. Due to space constraints, we include additional related work in the Appendix.
Interpreting Deep Generative Models. Several works have proposed methods for decomposing the latent space of a generative model into interpretable or disentangled directions. Early work included Beta-VAE (Higgins et al., 2017), which modified the variational ELBO in the original VAE formulation, and InfoGAN (Chen et al., 2016), which maximized the mutual information between a subset of the latent code and the generated data. Later work has sought to disentangle factors of variation by mixing latent codes (Hu et al., 2018), adding additional adversarial losses (Mathieu et al., 2016), and using contrastive learning (Ren et al., 2021).
Our work follows a recent line of research that looks for structure in large, pretrained generative models. Shen & Zhou (2020) perform a direct decomposition of model weights to find disentangled
directions, while Peebles et al. (2020) penalize nonzero second-order interactions between different latent dimensions, and Voynov & Babenko (2020) find interpretable directions by introducing an additional reconstruction network.
Differently from the works above, we conduct a deep study of one specific type of structure (foreground/background separation) encoded in the latent space. Other works have taken this approach in the context of extracting 3D structure from 2D images; for example, IG-GAN (Lunz et al., 2020) uses a neural renderer to recover 3D (voxel-based) representations of scenes, and GAN2Shape (Pan et al., 2021) exploits viewpoint and lighting variations in generated images to recover 3D shapes.
Unsupervised Object Segmentation. Prior work on unsupervised object segmentation can be divided into two categories: those that employ generative models to obtain segmentation masks and those that employ purely discriminative methods such as contrastive learning (Ji et al., 2019a; Ouali et al., 2020). Here, we focus on generative approaches.
Nearly all generative approaches are based on the idea of decomposing the generative process in a layer-wise fashion; in general, the foreground and background of an image are generated separately and then combined to obtain a final image. Specifically, ReDo (Chen et al., 2019) trains a generator to re-draw new objects on top of old objects, and enforces realism through adversarial training. (Bielski & Favaro, 2019) generates a background, a foreground, and a foreground mask separately and composite them together; they prevent degenerate outputs (i.e. the foreground and background being the same) by randomly shifting the foreground relative to the background. CopyPaste GAN (Arandjelović & Zisserman, 2019) receives two images as input and copies parts of one image onto the other. OneGAN (Benny & Wolf, 2020) learns to simultaneously generate, cluster, and segment images with a combination of GANs, VAEs, and additional encoders. Equivariant Layered GAN (Yang et al., 2021) first trains a new layerwise GAN and then trains a segmentation network on synthetic data. Labels4Free (Abdal et al., 2021) proposes a new layerwise network inspired by StyleGAN for foreground-background segmenation. Common difficulties with these layer-wise approaches above include the challenges of training new GANs and scaling beyond simple datasets (e.g. CUB, Flowers).
One work along different lines is by Voynov et al. (2021), which uses a pretrained BigBiGAN (Donahue & Simonyan, 2019) generator rather than proposing a new layer-wise GAN. Voynov et al. (2021) use the method from Voynov & Babenko (2020) to decompose the latent space into interpretable directions. A recent version of their method also picks out this direction without supervision using the idea that a foreground-background separating direction should be decomposable into two affine operators acting on different sets of pixels.
Our approach is based on generative modeling, but it differs from other approaches in that we seek a general-purpose method to find foreground/background structure implicitly encoded in standard, non-layer-wise GANs rather than encoding it explicitly or searching for it manually. This enables us to leverage any of the numerous existing generators that have already been pretrained on millions of high-resolution images, rather than developing a new GAN architecture for this specific task. Differently from layer-wise approaches, our approach does not require training new GANs and it does not rely on the assumption that the foreground and background of an image are independent. Differently from Voynov et al. (2021), we show through experiments that our method can be applied to a wide range of different GANs and find that it delivers superior performance on four out of five object segmentation and saliency detection benchmarks. Finally, unlike any of these previous works, we demonstrate that our foreground-background segmentation network can be used as a drop-in replacement for saliency networks for the task of learning dense semantic image representations. By clustering these representations, we are able to extend our method from foregound-background segmentation to semantic segmentation.

heading: 3 METHOD
text: Let x ∈ R3×H×W be a (color) image. A generator (network) is a function G : RD → R3×H×W that maps code variables z to images x = G(z). Optionally, some generative models come with an encoder function E : R3×H×W → RD which computes an approximate inverse of the generator (i.e. G(E(x)) ≈ x). A challenge in generating images is that individual pixels exhibit complex correlations, caused by the fact that the images are obtained as the composition of a number of different objects. For example,
all pixels that belong to a dog have a similar color, characteristic of dog’s instance. However, the correlation is much less strong between pixels that belong to different objects. This is because, while object in a scene are not entirely independent, their correlation is much weaker than within the structure of objects.
Intuitively, an image generator must learn to account for such correlations in order to generate realistically-looking images. In particular, we expect the generator to capture the idea that pixels that belong to the same object have a related appearance, whereas the appearance of pixels that belong to different objects or, as it may be, to a foreground object and its background, should be much more statistically independent.
Given a generator function G, it is then natural to ask whether such correlations can be extracted and used not just for the purpose of generating images, but also for analyzing them. In order to explore this idea, we consider perturbing the code z via a small increment v ∈ RD, where ∈ R and v ∈ SD−1 is a unit vector. Because the dimension D of the embedding space is typically much smaller than the dimension 3HW of the generated images, codes provide highly-compressed views of the data (for example, D = 120 for BigBiGAN (Donahue & Simonyan, 2019) and the self-conditioned GAN). As such, most changes in the code are likely to affect most if not all pixels in the image. However, if the generator did in fact learn to compose objects, then one could hope too find specific variations v that only affect only portions of the image, corresponding to individual objects, and use the latter to highlight and segment them.
Empirically, we find that the situation is not as simple. Specifically, it is not easy to find changes in the code that leave part of the pixels exactly constant while changing other pixels. However, we find that there are directions that affect foreground and background regions in a systematic and characteristic manner. Furthermore, we show that these directions are ‘universal’, in the sense that the same v works for all codes z, and are thus characteristic of a given generator network G.

heading: 3.1 FINDING INFORMATIVE CODE VARIATIONS
text: Next, we introduce an automated criterion to select informative changes v in code space. To this end, we consider an image x = G(z) generated from a random code z ∼ Z , where Z is the code distribution characteristic of the generator (e.g. an i.i.d. Gaussian). We then consider a modified image x′ = G(z + v) and observe the change x→ x′. We compare the two images using two criteria. The first one preserves the structure of the image x. We capture the latter by imposing that x and x′ generate approximately the same edges when fed to a simple edge detector. The intuition is that we wish v to affect the appearance of objects without changing their shape. By preventing objects from ‘moving around’ the image or deforming, we make it significantly easier to extract an image segmentation from the change x→ x′. This loss takes the form:
Ls(v) = 1
N N∑ i=1 ‖S(G(zi + v))− S(G(zi))‖2
Perturbation Amount ( 𝜺 ) 0.0 2.0 4.0 8.0-8.0 -4.0 -2.0
(a) A comparison of generated images for different values of the perturbation length , using the BigBiGAN generator. A value of = 0.0 corresponds to the original image, with a random Gaussian latent vector z ∼ N (0, 1).
Original Image Foreground Darker Foreground Lighter Mask ( 𝒗l )Mask ( 𝒗b ) Mask ( 𝒗l and 𝒗b ) Original
Image Foreground Darker Foreground Lighter Mask ( 𝒗l )Mask ( 𝒗b ) Mask
( 𝒗l and 𝒗b )
(b) A comparison of perturbed images and their masks for vb (foreground darker), vl, (foreground lighter), and the combination vb and vl. Using both directions yields visually superior segmentations.
where zi ∼ Z and S is the Sobel-Feldman operator:
[S(x)]ij = 3∑ c=1 (g ∗ xc::)2ij + (g> ∗ xc::)2ij ,
and g = [1 2 1] · [1 0 −1]> . This loss encourages x and x′ to be similar. We thus also need a loss that encourages the direction v to explore a non-zero change of the image. We consider an image contrast variation and additionally exploit the photographer bias, that objects are often placed in the middle of the image. This is captured by the loss:
Lc(v) = 1
N N∑ i=1 3∑ c=1 〈G(z + v), r〉
where r ∈ RH×W is a ‘radial’ prior:
rij = 1− 1
α
√( i− H + 1
2
)2 + ( j − W + 1
2 )2 with normalization factor α = 14 √ (H − 1)2 + (W − 1)2 that linearly interpolates from 1 in the center of the image to −1 at the boundary. This encourages finding a direction v that changes the brightness in the center of the image opposite to the border. In order to learn v, the two losses are combined with a weighting factor λ.
L(v) = λLc(v) + Ls(v) (1)
Given this fully-automatic procedure, the latent code direction v may be thought of as a function of the generator G and the weighting factor λ.

heading: 3.1.1 COMBINING INFORMATIVE CODES
text: Optimizing Eq. (1) with λ > 0 encourages the network to produce a shift v that brightens the foreground and darkens the background of an image. However, there is no constraint that λ need be positive; by negating λ and optimizing a second time, we obtain another direction v that shifts the foreground dark and the background light.
Although using only one direction suffices for our method, we find that we can improve performance by using both. As a result, for the remainder of the paper, let vl represent the direction that shifts the foreground lighter, and vd represent the direction that shifts the foreground darker.

heading: 3.2 LEARNING A SEGMENTATION MODEL
text: Once the latent directions vd and vl have been found, the process of extracting a segmentation mask is straightforward: we label as foreground regions the pixels in which the image generated with the
foreground-lighter shifted latent code is lighter than the image generated with the foreground-darker shifted latent code. That is, for a generated image x = G(z), we have:
M(z) = sign(G(z + vl)−G(z + vd)) (2)
Alternatively, if we use only a single direction vl or vd, M(z) is set to either:
G(z + vl)−G(z) or G(z)−G(z + vd) (3)
Given the learned direction v, we use it to generate a training set as follows:
D = {(G(zi),M(zi)) : zi ∼ Z, i = 1, . . .}.
This dataset may then be used to train any dense segmentation network Ψ (i.e., a UNet (Ronneberger et al., 2015)) in the standard fashion. That is, we minimize the pixel-wise binary cross-entropy loss between the segmentation output Ψ(G(z)) ∈ RH×W and the (synthesized) mask M(z):
L(Ψ|z) = − 1 HW ∑ u∈[H]×[W ] log p(sign(Mu(z))|Ψu(G(z)))
where p(m|s) = mσ(s)+(1−m)σ(−s), u is a pixel index and sign is the sign function. Unlike previous object segmentation methods, our method requires no additional losses or constraints to ensure the stability of training. By sampling z, we can generate an ‘infinite’ dataset for learning the network Ψ. Although we described the procedure above for unconditional GANs, our method applies just as well for weakly-supervised conditional GANs, where the generator G(z, y) also depends on a class label; we simply sample a label y uniformly at random for each generated image.

heading: 3.2.1 REFINING THE GENERATED DATASET
text: An advantage of training with GAN-generated data is that the dataset size is infinite, which means that one is free to curate one’s dataset and discard uninformative training examples. In our case, we found that it was helpful to refine the dataset by (1) discarding images with masks that were too large, (2) discarding images for which the latent code shift did not produce a significant change in brightness, and (3) removing small connected components from the mask. The exact details are given in the Supplementary Material.

heading: 4 EXPERIMENTS
text: In this section, we present an extensive set of experiments demonstrating the method’s performance, its wide applicability across image datasets, and its generalizability across GAN architectures.

heading: 4.1 EXPERIMENTAL SETUP
text: As our method is generator-agnostic, we apply our method to twelve generators, including three unconditional and nine conditional GANs. For the three unconditional GANs (BigBiGAN (Donahue & Simonyan, 2019), SelfCondGAN (Liu et al., 2020), and UncondGAN (Liu et al., 2020)), our procedure is completely unsupervised. For conditional GANs, our method is still unsupervised but the GAN naturally relies on class supervision for training.
To demonstrate the efficacy of our method across resolutions and datasets, we implement both, GANs trained on ImageNet (Deng et al., 2009) at a resolution of 128px, and GANs trained on the smaller TinyImageNet dataset (100,000 images split into 200 classes) at a resolution of 64px. All experiments performed across all GANs utilize the same set of hyperparameters for both optimization and segmentation. This is a key advantage of our method relative to other unsupervised/weaklysupervised image segmentation methods (Chen et al., 2019; Bielski & Favaro, 2019; Benny & Wolf, 2020; Arandjelović & Zisserman, 2019), which are sensitive to dataset-specific hyperparameters.

heading: 4.2 EVALUATION DATA
text: We evaluate the performance of our model on three standard saliency detection benchmarks (DUTOmrom (Yang et al., 2013), DUTS (Wang et al., 2017), ECSSD (Shi et al., 2016)) and two standard
object segmentation benchmarks (CUB (Welinder et al., 2010), and Flowers-102 (Nilsback & Zisserman, 2009)). For the saliency datasets, we evaluate using (pixel-wise) accuracy, mean intersectionover-union (IoU), and Fβ-score with β2 = 0.3. For the object segmentation datasets, we evaluate using accuracy, IoU, and maxFβ (the maximum Fβ score over a range of 255 uniformly distributed binarization thresholds between 0 and 1).

heading: 4.3 RESULTS
text: Performance on Benchmarks. In Table 1, we compare our method to other recent work. We emphasize that our method uses the same model for all datasets and has not seen any of the (training or evaluation) data for these datasets before. Our method delivers strong performance across datasets, approaching and even outperforming some supervised/handcrafted saliency detection methods. In comparison to Voynov et al. (2021), we perform better on four out of five benchmarks (all except CUB), even though we do not rely on humans to hand-pick latent directions. In comparison to layerwise GANs (Chen et al., 2019; Bielski & Favaro, 2019; Benny & Wolf, 2020; Arandjelović & Zisserman, 2019), we perform similarly on CUB and Flowers-102, but we cannot compare our
method with layerwise GANs on complex datasets (e.g. DUTS) because they do not produce any meaningful results. Due to the difficulty of training GANs, they are only ever trained on datasets consisting of images from a single domain with a single main subject, such as birds or flowers. By contrast, our ability to leverage pre-trained generators means that our method scales to complex and diverse datasets, such as those used for saliency detection.
Performance across Generators. We investigate the generality of our method by performing the same optimization and training pipeline with twelve different GANs. For each generator, we optimize to obtain a latent direction vl, train a segmentation model using this direction, and evaluate its performance across the five datasets above. The same hyperparameters are kept constant for all GANs, including λ = 5.0 during the optimization phase.
Results are shown in Table 2; BigBiGAN performs best, but all networks, even those using relatively weak TinyImageNet-trained GANs (e.g., GGAN (Lim & Ye, 2017)), deliver reasonable segmentation performance. This highlights the benefits of our fully-automatic segmentation pipeline; our method performs well
across a wide range of generators trained on different datasets at different resolutions.
Naturally, the quality of a final segmentation network produced by our method is related to the quality of the underlying generator. Figure 4 plots the Frechet Inception Distance (FID) score of nine conditional GANs versus the average accuracy of the corresponding segmentation networks produced by our method. Lower FID scores, which correspond to better GANs, correlate with improved accuracy. This correlation suggests that as better GANs architectures are developed, our method will continue to produce better unsupervised segmentation networks.
Ablation: Comparing latent directions. We compare the performance of a segmentation masks using the two latent directions vd and vl together (Eq. (2)), or each of them individually (Eq. (3)) visually in Fig. 3a. In Table 6, we quantitatively compare the results of these three methods along with a fourth method in which we ensemble the final segmentation networks produced by vd and vl individually. The foreground-lighter (vl) and foreground-darker (vd) directions yield similar results when used individually. The combination (vl and vd) provides superior results, on par with the ensemble. Unlike the ensemble, which requires training two networks, the combination of vd and vl adds minimal overhead compared to training with one direction.
Ablation: Varying λ, , the Central Prior, Random Initializations, Amount of Generated Data. The two hyperparameters in the optimization stage of our method are λ, which controls the trade-off between brightness and consistency, and , which controls the magnitude of the perturbation. We find that the process is only modestly sensitive to changes in these hyperparameters. We also find that using the central prior compared to a spatially-agnostic loss term is better, but only moderately. Detailed numerical results, including the ablation on the central prior, are given in Section A.3. In Appendix B.3, we also investigate how segmentation performance varies with the number of generated images, finding a clear log-linear relationship.
Qualitative Results. By inspection, we find that our optimization procedure is able to edit images in such a manner that the foreground becomes lighter and the background becomes darker. In numerous cases, the network appears to convert the scene from daytime to nighttime. Furthermore, better GANs generally produce qualitativley better segmentations. Please refer to Appendix A and Section A.1.3 for illustrations.

heading: 4.4 EXTENSION TO SEMANTIC SEGMENTATION
text: Finally, we demonstrate that our network may be extended from binary foreground-background segmentation to semantic segmentation, the task of assigning each pixel in an image into one of K
semantic categories. Due to the challenging nature of this task, it has not been attempted by any previous works in the GAN-based segmentation space.
We extend our method by following the dense contrastive learning approach proposed by Van Gansbeke et al. (2021). In this approach, binary masks are extracted from a set of images using a foreground-background segmentation model, sometimes called a “mid-level visual prior.” Then, a network is trained to generate pixel-wise features using a mask-based contrastive loss: features corresponding to pixels in the foreground of the image are pulled toward the features of other pixels in the mask and pushed away from the features of background pixels. After training, semantic segmentations may be extracted by clustering these pixel-wise features across an entire dataset.
Importantly, Van Gansbeke et al. (2021) uses saliency detection networks to generate their object masks. Although these saliency detectors are sometimes called “unsupervised,” they are actually initialized using pretrained semantic segmentation networks1. We propose to use our object segmentation network as a drop-in replacement for these saliency networks, making the entire process entirely unsupervised. We leave the rest of their method (i.e. the dense contrastive learning and the evaluation procedure) unchanged.
We perform experiments on the PASCAL VOC dataset, which contains 20 semantic classes. Experimental details are included in Section A.1.3 and K-Means clustering results are shown in Table 3. We compare to a range of baselines, along with two models from Van Gansbeke et al. (2021) using different levels of supervised pretraining. Our network is competitive with Van Gansbeke et al. (2021) despite being entirely unsupervised. Furthermore, in Table 4 in Section A.1.3 we show that this procedure works for a wide range of GANs, demonstrating the generalizability of our approach.

heading: 5 CONCLUSIONS
text: We find that extracting a salient object segmentation from the latent space of a GAN is not only possible without supervision but also leads to state-of-the-art unsupervised segmentation performance on several benchmark datasets. In contrast to existing methods that have been engineered specifically for this task, we extract segmentations from a network trained for a very different purpose — generating images. Surprisingly, we are able to generalize to a wide range of segmentation benchmarks without directly training on any real images, and even extend our results to semantic segmentation. Our findings directly prompt future research questions about what other concepts of the physical world can be automatically extracted from generative models, and to what extent we can use such extracted concepts to replace human supervision in other computer vision tasks.
1In the case of Van Gansbeke et al. (2021), they use DeepUSPS(Nguyen et al., 2019), which is initialized using a supervised network pretrained for semantic segmentation on Cityscapes.

heading: 6 REPRODUCIBILITY STATEMENT
text: We aim to ensure that our experiments are entirely and easily reproducible. We upload code to the Supplementary Material to fully reproduce all experiments. This code contains a README file with a detailed description of the code structure, which should help enable others to reproduce and later extend upon our work. We also take care to describe all hyperparameters and implementation details in the Appendix. Our results do not require extremely large amounts of compute; they can be reproduced with a single GPU by researchers with computational constraints.

heading: 7 ETHICS STATEMENT
text: It is important to discuss the potential ethical issues involved with training large-scale generative models and segmentation networks along the lines proposed by our paper.
First of all, the task of segmentation is predicated upon classifying objects into predetermined (either binary or semantic) categories; the definition of these categories, especially in the case of semantic segmentation, may introduce biases into the task itself. Second, when training models on largescale datasets, it is essential to consider the biases and data privacy issues introduced in the data collection process. For example, the PASCAL-VOC dataset, which we use for the task of semantic segmentation, is composed of images scraped from Flickr. As a result, it is composed primarily of photographs from the United States and Europe, and the “person” class contains primarily images of white individuals. Additionally, it is not clear whether the individuals in these photographs consent to being used to train image segmentation models, bringing up the issue of data privacy.
From an ethical perspective, our method is slightly different from standard segmentation models because it is trained solely on GAN-generated images; this is not to say that it is ethically better or worse, but that it involves different ethical considerations. On the one hand, this might alleviate some data privacy concerns, as the segmentation training data is synthetic. However, since this training data is generated by a GAN, one has to examine the data and methodology originally used to pretrain the GAN; any biases present in this data will likely be reproduced or amplified by the GAN . For example, if one uses a GAN trained on ImageNet to perform object segmentation, it may perform better on white individuals than individuals of other races due to the disproportionate percentage of white individuals in the training data (Steed & Caliskan, 2021). Investigating biases introduced by GANs remains an active area of research in the machine learning community (Jain et al., 2020; Tan et al., 2020), and these ethical discussions extend to our GAN-based segmentation method.

heading: A APPENDIX
text: A.1 IMPLEMENTATION DETAILS

heading: A.1.1 OPTIMIZATION
text: First, we optimize for the latent code vectors vd and vl. We generate latent codes z ∼ N (0, 1) and optimize the vector vl (or vd) by gradient descent with the Adam (Kingma & Ba, 2014) optimizer and learning rate 0.05. We use λ = 5 for the light direction vl and λ = −5 for the dark direction vb. We perform 1000 optimization steps, by which point vl (or vd) has converged.

heading: A.1.2 OBJECT SEGMENTATION
text: To generate data, we sample z ∼ N (0, 1), produce the mask m = M(z), and refine the mask as described in the main paper. We use a simple UNet (Ronneberger et al., 2015) with bilinear down/upsampling as our segmentation network. Following (Voynov et al., 2021), we train for 12000 steps using Adam with learning rate 10−3 and batch size 95, decaying the learning rate by a factor of 0.2 at iteration 8000.

heading: A.1.3 SEMANTIC SEGMENTATION
text: We follow the procedure from (Van Gansbeke et al., 2021), but we use binary masks produced by our object segmentation model as a drop-in replacement for their saliency detection masks. Under this procedure, we begin by applying our segmentation network to extract masks from the trainaug split of PASCAL-VOC (10582 images). We then train a dilated ResNet-50 model with DeepLabv3 head, initialized with a pretrained MoCo ResNet-50. As in (Van Gansbeke et al., 2021), this network has a main head and an auxiliary head: the main head predicts dense pixel-wise features of dimension 32, and the auxiliary head predicts the object segmentation mask for the purpose of regularization. We train the output of the main head using a pixel-wise version of the InfoNCE loss. Specifically, for a feature zi corresponding to a pixel i in the foreground mask Mi, we have:
Li = − log exp(zi · z+/τ)∑K k=0 exp(zi · z−/τ)
where z+ is the average embedding of other pixels in the mask Mi, and z− is the average embeding of pixels in the background:
z+ = 1 |Mi| ∑ i∈Mi zi, z− = 1 N − |Mi| ∑ i 6∈Mi zi
where N is the number of pixels in the image.
We use the same hyperparameters as (Van Gansbeke et al., 2021) for the contrastive learning procedure. We train for 40 epochs using random crops of size 224, learning rate 0.01, and output dimension 32. After training, we evaluate with K-Means clustering with 21 clusters (20 semantic categories and 1 background category). These clusters are computed globally over the entire validation set and matched to the ground truth labels using the Hungarian algorithm.
We repeat this same procedure identically for a large range of GANs. Figure 9 compares the result of final segmentation networks for different GANs: all images produce sensible segmentations, with better GANs generally producing better segmentations.

heading: A.2 DATASETS
text: Below, we give statistics of our evaluation datasets. Note that these are only used for evaluation, as only GAN-generated images are seen during training:

heading: Dataset Num. Images Type Crop

heading: A.3 ADDITIONAL ABLATIONS
text: Ablation: Comparing latent directions. In addition to comparing the networks resulting from vl and vb, we also compare the actual latent directions vl and vb. Due to the nonlinearity of the generator function, the optimal unit directions vl and vd are not necessarily negations of one another; indeed, we found in practice that they were close to but not exactly antiparallel. Table 9 in Section A.3 gives exact numbers for a variety of different generator architectures.
Ablation over λ. The most notable hyperparameter in the optimization stage of our method is λ, which controls the trade-off between brightness and consistency. Table 7 compares segmentation results for BigBiGAN with λ = 10, 5, 2.5, and 1.25, with 5 performing best.
Ablation over . In Table 8, we show ablation results for changing during the optimization process. Note that since the GAN used in this set of experiments (BigBiGAN) has a 120-dimensional latent space, the distribution of the norm of the N (0, 1) latent vectors used to train the GAN is concentrated around (approximately) 11. That is to say, a shift of magnitude = 6 in the latent space is very large.
Ablation: Random initialization. Given the importance of the latent direction optimization procedure in our segmentation pipeline, it is natural to ask whether different initializations yield materially different directions in the latent space. To answer this question, we optimized ten vectors v on
CUB Flowers DUT-OMRON DUTS ECSSD Acc IoU Acc IoU Acc IoU Acc IoU Acc IoU
CUB Flowers DUT-OMRON DUTS ECSSD Acc IoU Acc IoU Acc IoU Acc IoU Acc IoU
BigBiGAN from different initializations and measured their pairwise cosine similarities. We found that the cosine similarities had mean 0.9866 with standard deviation 0.0044, which is extremely small relative to the values in Table 9. Thus, the random initialization of the optimization procedure does not have a large impact on the final direction.
Ablation: On the necessity of the center prior. Here, we conduct a small experiment to investigate the necessity of the central prior in the optimization step of our segmentation pipeline. We replace it with a spatially-agnostic loss term with no central prior: it simply encourages the shifted image to have high variance in brightness. We find that this spatially-agnostic loss still yields good results, although not quite as good as the spatial prior loss. Numerical results are shown in Table 10.

heading: A.4 ADDITIONAL RELATED WORK
text: Saliency Detection. Object segmentation is closely related to saliency detection, the problem of finding significant (salient) objects in an image. Although most saliency detection models are trained with pixel-level supervision (or human eye movement data), the past few years have seen some research into unsupervised/weakly-supervised saliency detection (Zhang et al., 2018; Nguyen et al., 2019; Zeng et al., 2019). These methods work by ensembling strong hand-crafted priors and distilling them into a deep network. In practice, they also initialize their networks with pretrained (supervised) image classifiers or semantic segmentation networks.
Learning from Synthetic Data. Finally, our method can be viewed from the perspective of learning from synthetic data. Motivated by the costly and time-consuming nature of data labeling, there has been a plethora of recent work on synthesizing, training with, and adapting from synthetic datasets. For example, one widely-studied line of research (Hoffman et al., 2018; Tsai et al., 2018; Zou et al., 2018; Vu et al., 2019; Toldo et al., 2020) tackles the task of semantic segmentation by training on data generated from video games (e.g. GTA5). With regard to adversarially-generated training data specifically (Shrivastava et al., 2017) uses a GAN-like network to enhance the realism of synthetic images while preserving label information.
Although we train our segmentation network using generated images only, we show in the experiments below that it generalizes to real-world images without the need for additional adaptation. Such additional adaptation could be an interesting avenue for future research.

heading: A.5 ADDITIONAL EXAMPLES

heading: A.5.1 EXAMPLES ACROSS DATASETS
text: In Figure 6, Figure 7,aand Figure 8, we show the results of applying our final segmentation network to random images from each of the five datasets on which we evaluated.

heading: A.5.2 EXAMPLES ACROSS GENERATORS
text: In Figure 9, we show examples of pairs of generated images and the corresponding extracted samples for a range of different GANs.

heading: A.5.3 SEMANTIC SEGMENTATION EXAMPLES
text: In Figure 10, we provide randomly selected examples of our semantic segmentations afterK-Means clustering on the PASCAL-VOC dataset. We see that our masks are of similar visual quality to those from MaskContrast (Van Gansbeke et al., 2021), despite the fact that our method is entirely unsupervised.
CUB Flowers DUT-OMRON DUTS ECSSD
Figure 7: Examples of the final segmentation network across evaluation datasets. From left to right: original image, ground truth, prediction.
CUB Flowers DUT-OMRON DUTS ECSSD
Figure 8: Examples of the final segmentation network across evaluation datasets. From left to right: original image, ground truth, prediction.

heading: Image Ground Truth MaskContrast Ours Image Ground Truth MaskContrast Ours

heading: B FURTHER EXPERIMENTS
text: Based on the helpful comments of our reviewers, we have conducted new experiments and created more visualizations to better understand how our method functions and performs.

heading: B.1 ADDITIONAL EXPERIMENTS ON FLOWERS
text: In this section, we provide a comprehensive explanation and analysis of our performance on the Flowers dataset.
The Flowers dataset is different from the other datasets investigated in our paper (CUB, DUTS, DUT-OMRON, ECSSD, PASCAL VOC) in that the “ground-truth” masks for Flowers were obtained using an automated segmentation method rather than human annotations. As a result, the ground truth are very unreliable. In fact, there are many instances in the Flowers dataset in which the ground truth mask is completely or nearly completely empty despite there clearly being a flower in the image. This issue is also mentioned by Chen et al. (2019), who claim that their method “provide[s] better masks” than the ground-truth in cases of disagreement.
Below, we present an additional experiment to improve our performance on Flowers. We switch the underlying GAN in our method from BigBiGAN (trained on ImageNet) to an unconditional StyleGAN2 trained on Flowers.
This experiment aims to show that our method works well on in-domain data (i.e. using a GAN trained on Flowers to segment Flowers) in addition to out-of-domain data (i.e. using a GAN trained on ImageNet to segment Flowers).
Specifically, we found publicly available weights on GitHub for an (unconditional) StyleGAN2 model Karras et al. (2019) trained on the Flowers dataset at a resolution of 256px, and we applied our method directly to this model. We use perturbation radius r = 0.2 rather than r = 2.0 as in BigBiGAN because the latent space of StyleGAN2 has a different scale. Apart from this singular parameter, however we apply our method without any hyperparameter tuning or additional modifications for the new StyleGAN2.
Results are shown in Table 11. We find that using StyleGAN2 dramatically improves the visual quality of our segmentations both quantitatively and qualitatively. Quantitatively, our performance under this setup is similar to other unsupervised methods which have been extensively finetuned for this task, such as Chen et al. (2019).
We show examples of failure cases in Fig. 13 and Fig. 14. The most severe failure cases involve empty ground-truth masks. Similarly to ReDo, we find that when our method and the ground truth disagree, our segmentations are often visually superior. At this level of performance, the high level of label noise means that exact mIoU/accuracy numbers are not meaningful. However, there is no doubt that switching generators to StyleGAN tremendously improves our flower segmentation performance.
Additionally, we sought to identify the extent to which this improvement was due purely to the increased resolution (256px) as opposed to the underlying GAN. We train a segmentation model 128px-downsampled versions of the StyleGAN2 generated images and show the results in Table 15. Higher resolution training improves performance modestly on the Flowers dataset, while the bulk of the improvement is due to the use of StyleGAN. This result supports our hypothesis that better GANs correspond to better unsupervised object segmentation performance under our model.

heading: B.2 ADDITIONAL EXPERIMENTS ON FACES
text: In this section, we provide additional experiments on the domain of face images.
This section is designed with comparison to Abdal et al. (2021) in mind. However, it is difficult to compare with Abdal et al. (2021) because they evaluate on non-standard datasets, they do not release any information about their data splits, and they do not release code or pretrained models. In particular, they conduct evaluations on CelebA-HQ-Mask using 1000 randomly selected images from the dataset, but they do not specify which images these are. Nevertheless, we tried to recreate their setup as best as possible to establish a comparison.
We applied our method to a StyleGAN2 trained on FFHQ. We use resolution 256px due to time constraints (and 1024px would likely improve results).
We use a perturbation radius r = 0.2, which is copied directly from the experiment in the section above (because both experiments use StyleGAN2). Again, with the exception of this single parameter, we did not change a single other hyperparameter in our entire learning setup to demonstrate the generalizability of our approach.
Results are shown in Table 12. Qualitatively, we find that our model tends to segment the hair and face but not the clothes, which is an equally valid segmentation but does not align with the ground truth where the person’s body is included in the foreground mask. This result could mean that StyleGAN2 internally models the clothing in a different manner from the face which would lead to this behavior in latent space.

heading: B.3 DATA SIZE
text: In this section, we investigate the relationship between the amount of synthetic training data and segmentation performance. All experiments in the main paper use 1,000,000 generated examples.
We train models for the same number of total iterations using a varying number of synthetic training images from 100 to 1,000,000. Results are shown in Table 14 and Fig. 11. We find consistently across all datasets that - similar to supervised learning - performance scales logarithmically with the dataset size. Differently from supervised learning, our labels are entirely free to generate, so we can scale our synthetic dataset as large as desired.

heading: B.4 SEMI-SUPERVISED LEARNING
text: In this section, we investigate whether our synthetic data can be used to augment existing supervised datasets in a semi-supervised learning setup.
We consider a subset of the CUB dataset with 1000 labeled examples along with synthetic data from our BigBiGAN (ImageNet-pretrained) model. Results are shown in Table 13.
First, we train a supervised model on the 1000 images with ground truth segmentations as a baseline. Similarly, we train a fully unsupervised model on our generated data alone. Now, in the semisupervised setting, we have access to 1000 labeled images. When training on the combination of both datasets we can already see an improvement over both baselines in IoU (which is the harder and less saturated metric). With access to some in-domain data, we can also take another step to reduce the domain gap between the generated and real images: we can filter our generated images by their distance to the supervised samples. Specifically, we select the 50000 nearest neighbors in our generated set (according to cosine similarity using a self-supervised ResNet-50 He et al. (2019)), and we refer to these as the “kNN generated images” in Table 13.
We see that a model trained on the combined ground truth and kNN samples surpasses both baselines and the supervised model. This result demonstrates that our pipeline can play a valuable role in semi-supervised object segmentation in addition to unsupervised object segmentation.

heading: B.4.1 FAILURE CASES
text: In this section, we describe common failure cases of our models.
Fig. 12, Fig. 13, Fig. 14, and Fig. 15 show examples of failure cases of our model.
One of the main failure cases of our method is to segment additional foreground objects beyond the “main object” in the image. For example, on the CUB dataset (Fig. 12), we often segment objects such as branches and bird feeders alongside the bird in the image. Across all datasets, one of the other main sources of errors is the object boundary that is not as precise as the object. This could potentially be improved in the future through the use of higher-resolution GANs or by postprocessing (e.g. with a CRF).
The Flowers dataset is a special case in which the largest source of error is label noise in the ground truth annotations. We discuss this issue extensively in the section on Flowers above. Fig. 13 shows our top 16 predictions by mIoU error in the Flowers dataset: the ground truth segmentations are
empty. Fig. 13 shows top failure cases on Flowers where the ground truth mask is not empty, which still mainly consists of low quality ground truth annotations. Occasionally, our method misses to segment a flower in the background or segments too much of the stem.
With regard to failure cases of mask generation, since there is no ground-truth associated with these generated images, these failure cases were found by manually filtering approximately 500 generated images (Fig. 15). Common failure cases include segmenting too much or too little of the image and are often attributable to low-quality images generated by the GAN. Thus, improvements in the generation quality of GANs and the performance of our method are closely linked.

Review summaries:
This paper describes a simple procedure that allows to train an unsupervised image segmentation model only using a generative model as input. By tricking the generator to predict proxy segmentation masks along with a generated image, the method allows to construct a possibly infinite dataset of images associated with a F/B mask. A segmentation model can then be trained on this dataset to predict foreground. The proposed approach is evaluated across a wide range of small-scale segmentation datasets, and compared with various approaches to the problem (both supervised, handcrafted and unsupervised). An adaptation of this approach to semantic segmentation is proposed and evaluated on Pascal VOC.

The paper tackles the problem of image segmentation. It focuses on finding foreground-background image separation cues from a pre-trained GAN and create a synthetic dataset. Once the dataset is created, it trains a segmentation models and test the approach on several benchmarks. The paper is centred around how to extract and detect this foreground-background separation in an unsupervised way. The method works with any GAN architecture and in the end achieves competitive results while being unsupervised.

The paper proposes an unsupervised approach for instance segmentation that leverages pretrained GANs. The approach is divided into two steps: first, the authors discover a latent direction in the generator that allows them to segment generated images. Then, the resulting image-label pairs are used to train a segmentation network. This network can finally be used to perform inference on unseen images. The approach is evaluated on a variety of GAN architectures. The main contribution claimed by the authors is that their method can be applied to off-the-shelf pretrained GANs, whereas competing approaches require architectural variations and re-training.

This paper presents a new unsupervised algorithm for image segmentation via deep generative models. Specifically, the authors combined edge preserving and brightness enhancement loss functions to search the latent manipulation directions which only adjusting the foreground object brightness. As a result, the directions could be utilized to synthesize image segmentation training samples. Experimental results on saliency detection and image segmentation demonstrate the effecteness of the proposed method.