Finding an Unsupervised Image Segmenter in each of your Deep Generative Models

1 INTRODUCTION . Recent years have seen rapid progress in the field of deep generative modeling of images , driven by a proliferation of research into Generative Adversaial Networks ( GANs ) ( Goodfellow et al. , 2014 ) . Nowadays , it is possible to generate high-resolution images of realistic objects and scenes ( Karras et al. , 2019 ; Brock et al. , 2019 ) . However , with the exception of generation for computer graphics , there has been limited research into how we might be able to leverage the representations learned by these powerful generative models to enhance other tasks , particularly those involving semantic understanding ( e.g . classification or segmentation ) . Generally , deep generative models learn to map latent codes to images , imposing simple statistical structures on the distribution of the latent codes , such as assuming an i.i.d . Gaussian distribution . Due to this structure , in some cases code dimensions acquire specific meanings which can be related to human-interpretable concepts ( e.g. , the rotation or size of an object ) ; however , the code space in high-quality generators ( e.g. , BigGAN ( Brock et al. , 2019 ) , BigBiGANs ( Donahue & Simonyan , 2019 ) , StyleGAN ( Karras et al. , 2019 ) ) is usually not easily interpretable . Nonetheless , it is intuitive that an efficient generative process should account for the structure of natural images , including for example the fact that images often comprise distinct foreground and background regions . In this paper , we validate this hypothesis by learning to separate foreground and background image regions from generator networks . Our approach starts from an arbitrary , off-the-shelf high-quality generator network trained on a large corpus of ( unlabeled ) images . While these generator networks are not explicitly trained for foreground/background segmentation , we show that such a separation emerges implicitly as a step to efficiently encode realistically-looking images . Specifically , we design a probing scheme that can extract such foreground/background information automatically , i.e . without manual supervision , from the generated images . This scheme works as follows ( cf . Fig . 1 ) . We start from a random code in latent space and learn a fixed , global offset that results in a change in the generated images . The offset is learned to alter the appearance of foreground and background such that a mask can be extracted from the changes in image space . The resulting masks provide segmentation maps for the generated images , but they can not yet be used to segment images from the real world . Given a natural image , the obvious approach would Under review as a conference paper at ICLR 2022 Top images were generated with seeds 5432 and 5431 G en er at ed im ag es an d ex tr ac te d m as ks Re al -w or ld im ag es a nd pr ed ic te d m as ks G en er at ed im ag es an d ex tr ac te d m as ks Re al -w or ld im ag es an d pr ed ic te d m as ks Figure 1 : We automatically find a universal latent direction in a GAN that can separate the foreground from the background without any supervision . We can then generate unlimited samples with masks to train a segmentation network that achieves state-of-the-art unsupervised segmentation performance . be to find the corresponding code in the latent space of the generator , and then obtain a mask with our method . Unfortunately , this inversion process is less than trivial . In fact , recent work provides strong evidence that the expressiveness of GANs is insufficient to encode arbitrary images ( Bau et al. , 2019 ) , meaning that the inversion problem has no solution in general . As we aim to build a general-purpose segmentation method , we take a different approach : we generate a labelled image dataset with foreground/background segmentations and use the generated dataset to train a standard segmentation network . With this , we show that our method can successfully learn accurate foreground-background segmentation networks with no manually provided labels at all . Differently from prior work in GAN-based image segmentation , we neither design a new GAN architecture specifically for the task of segmentation nor use manual supervision to extract segmentation information from an existing GAN . Thus , we can discover meaningful latent directions for any GAN with no need for model-specific manual intervention . Extensive experiments on five segmentation datasets across twelve different GANs demonstrate the effectiveness and generalizability of our approach . Moreover , by constructing our image segmenters from generator networks trained on a generic large-scale datasets such as ImageNet , our method can learn to generically segment objects from a wide range of visual domains . Specifically , when we apply our generic segmenters to the CUB200 ( Welinder et al. , 2010 ) and Oxford Flowers ( Nilsback , 2009 ) datasets , we attain very strong foreground-background results despite not training on this data . Similarly , when we apply our generic segmenters to three saliency detection benchmarks , our method approaches and sometimes even exceeds the performance of supervised and handcrafted saliency detection methods . An analysis of our results also shows that segmentation performance directly correlates with the quality of the underlying GAN , suggesting that foreground/background separation is an important concept in learning generative models . Finally , we demonstrate that our method may be used as a drop-in replacement for saliency networks for the purpose of learning pixel-wise semantic image representations . These pixel-wise representations can then be clustered to obtain fully unsupervised semantic segmentations , extending our method beyond foreground-background segmentation . Thus , we not only demonstrate for the first time that it is possible to obtain semantic segmentations using GANs , but also that this information may be extracted from a wide range of generic GANs trained on general-purpose image datasets . 2 RELATED WORK . Below , we describe how our method relates to recent work in interpreting generative models and object segmentation . Due to space constraints , we include additional related work in the Appendix . Interpreting Deep Generative Models . Several works have proposed methods for decomposing the latent space of a generative model into interpretable or disentangled directions . Early work included Beta-VAE ( Higgins et al. , 2017 ) , which modified the variational ELBO in the original VAE formulation , and InfoGAN ( Chen et al. , 2016 ) , which maximized the mutual information between a subset of the latent code and the generated data . Later work has sought to disentangle factors of variation by mixing latent codes ( Hu et al. , 2018 ) , adding additional adversarial losses ( Mathieu et al. , 2016 ) , and using contrastive learning ( Ren et al. , 2021 ) . Our work follows a recent line of research that looks for structure in large , pretrained generative models . Shen & Zhou ( 2020 ) perform a direct decomposition of model weights to find disentangled directions , while Peebles et al . ( 2020 ) penalize nonzero second-order interactions between different latent dimensions , and Voynov & Babenko ( 2020 ) find interpretable directions by introducing an additional reconstruction network . Differently from the works above , we conduct a deep study of one specific type of structure ( foreground/background separation ) encoded in the latent space . Other works have taken this approach in the context of extracting 3D structure from 2D images ; for example , IG-GAN ( Lunz et al. , 2020 ) uses a neural renderer to recover 3D ( voxel-based ) representations of scenes , and GAN2Shape ( Pan et al. , 2021 ) exploits viewpoint and lighting variations in generated images to recover 3D shapes . Unsupervised Object Segmentation . Prior work on unsupervised object segmentation can be divided into two categories : those that employ generative models to obtain segmentation masks and those that employ purely discriminative methods such as contrastive learning ( Ji et al. , 2019a ; Ouali et al. , 2020 ) . Here , we focus on generative approaches . Nearly all generative approaches are based on the idea of decomposing the generative process in a layer-wise fashion ; in general , the foreground and background of an image are generated separately and then combined to obtain a final image . Specifically , ReDo ( Chen et al. , 2019 ) trains a generator to re-draw new objects on top of old objects , and enforces realism through adversarial training . ( Bielski & Favaro , 2019 ) generates a background , a foreground , and a foreground mask separately and composite them together ; they prevent degenerate outputs ( i.e . the foreground and background being the same ) by randomly shifting the foreground relative to the background . CopyPaste GAN ( Arandjelović & Zisserman , 2019 ) receives two images as input and copies parts of one image onto the other . OneGAN ( Benny & Wolf , 2020 ) learns to simultaneously generate , cluster , and segment images with a combination of GANs , VAEs , and additional encoders . Equivariant Layered GAN ( Yang et al. , 2021 ) first trains a new layerwise GAN and then trains a segmentation network on synthetic data . Labels4Free ( Abdal et al. , 2021 ) proposes a new layerwise network inspired by StyleGAN for foreground-background segmenation . Common difficulties with these layer-wise approaches above include the challenges of training new GANs and scaling beyond simple datasets ( e.g . CUB , Flowers ) . One work along different lines is by Voynov et al . ( 2021 ) , which uses a pretrained BigBiGAN ( Donahue & Simonyan , 2019 ) generator rather than proposing a new layer-wise GAN . Voynov et al . ( 2021 ) use the method from Voynov & Babenko ( 2020 ) to decompose the latent space into interpretable directions . A recent version of their method also picks out this direction without supervision using the idea that a foreground-background separating direction should be decomposable into two affine operators acting on different sets of pixels . Our approach is based on generative modeling , but it differs from other approaches in that we seek a general-purpose method to find foreground/background structure implicitly encoded in standard , non-layer-wise GANs rather than encoding it explicitly or searching for it manually . This enables us to leverage any of the numerous existing generators that have already been pretrained on millions of high-resolution images , rather than developing a new GAN architecture for this specific task . Differently from layer-wise approaches , our approach does not require training new GANs and it does not rely on the assumption that the foreground and background of an image are independent . Differently from Voynov et al . ( 2021 ) , we show through experiments that our method can be applied to a wide range of different GANs and find that it delivers superior performance on four out of five object segmentation and saliency detection benchmarks . Finally , unlike any of these previous works , we demonstrate that our foreground-background segmentation network can be used as a drop-in replacement for saliency networks for the task of learning dense semantic image representations . By clustering these representations , we are able to extend our method from foregound-background segmentation to semantic segmentation . 3 METHOD . Let x ∈ R3×H×W be a ( color ) image . A generator ( network ) is a function G : RD → R3×H×W that maps code variables z to images x = G ( z ) . Optionally , some generative models come with an encoder function E : R3×H×W → RD which computes an approximate inverse of the generator ( i.e . G ( E ( x ) ) ≈ x ) . A challenge in generating images is that individual pixels exhibit complex correlations , caused by the fact that the images are obtained as the composition of a number of different objects . For example , all pixels that belong to a dog have a similar color , characteristic of dog ’ s instance . However , the correlation is much less strong between pixels that belong to different objects . This is because , while object in a scene are not entirely independent , their correlation is much weaker than within the structure of objects . Intuitively , an image generator must learn to account for such correlations in order to generate realistically-looking images . In particular , we expect the generator to capture the idea that pixels that belong to the same object have a related appearance , whereas the appearance of pixels that belong to different objects or , as it may be , to a foreground object and its background , should be much more statistically independent . Given a generator function G , it is then natural to ask whether such correlations can be extracted and used not just for the purpose of generating images , but also for analyzing them . In order to explore this idea , we consider perturbing the code z via a small increment v ∈ RD , where ∈ R and v ∈ SD−1 is a unit vector . Because the dimension D of the embedding space is typically much smaller than the dimension 3HW of the generated images , codes provide highly-compressed views of the data ( for example , D = 120 for BigBiGAN ( Donahue & Simonyan , 2019 ) and the self-conditioned GAN ) . As such , most changes in the code are likely to affect most if not all pixels in the image . However , if the generator did in fact learn to compose objects , then one could hope too find specific variations v that only affect only portions of the image , corresponding to individual objects , and use the latter to highlight and segment them . Empirically , we find that the situation is not as simple . Specifically , it is not easy to find changes in the code that leave part of the pixels exactly constant while changing other pixels . However , we find that there are directions that affect foreground and background regions in a systematic and characteristic manner . Furthermore , we show that these directions are ‘ universal ’ , in the sense that the same v works for all codes z , and are thus characteristic of a given generator network G .

Summary: This paper describes a simple procedure that allows to train an unsupervised image segmentation model only using a generative model as input. By tricking the generator to predict proxy segmentation masks along with a generated image, the method allows to construct a possibly infinite dataset of images associated with a F/B mask. A segmentation model can then be trained on this dataset to predict foreground. The proposed approach is evaluated across a wide range of small-scale segmentation datasets, and compared with various approaches to the problem (both supervised, handcrafted and unsupervised). An adaptation of this approach to semantic segmentation is proposed and evaluated on Pascal VOC.