Finding an Unsupervised Image Segmenter in each of your Deep Generative Models

1 INTRODUCTION . Recent years have seen rapid progress in the field of deep generative modeling of images , driven by a proliferation of research into Generative Adversaial Networks ( GANs ) ( Goodfellow et al. , 2014 ) . Nowadays , it is possible to generate high-resolution images of realistic objects and scenes ( Karras et al. , 2019 ; Brock et al. , 2019 ) . However , with the exception of generation for computer graphics , there has been limited research into how we might be able to leverage the representations learned by these powerful generative models to enhance other tasks , particularly those involving semantic understanding ( e.g . classification or segmentation ) . Generally , deep generative models learn to map latent codes to images , imposing simple statistical structures on the distribution of the latent codes , such as assuming an i.i.d . Gaussian distribution . Due to this structure , in some cases code dimensions acquire specific meanings which can be related to human-interpretable concepts ( e.g. , the rotation or size of an object ) ; however , the code space in high-quality generators ( e.g. , BigGAN ( Brock et al. , 2019 ) , BigBiGANs ( Donahue & Simonyan , 2019 ) , StyleGAN ( Karras et al. , 2019 ) ) is usually not easily interpretable . Nonetheless , it is intuitive that an efficient generative process should account for the structure of natural images , including for example the fact that images often comprise distinct foreground and background regions . In this paper , we validate this hypothesis by learning to separate foreground and background image regions from generator networks . Our approach starts from an arbitrary , off-the-shelf high-quality generator network trained on a large corpus of ( unlabeled ) images . While these generator networks are not explicitly trained for foreground/background segmentation , we show that such a separation emerges implicitly as a step to efficiently encode realistically-looking images . Specifically , we design a probing scheme that can extract such foreground/background information automatically , i.e . without manual supervision , from the generated images . This scheme works as follows ( cf . Fig . 1 ) . We start from a random code in latent space and learn a fixed , global offset that results in a change in the generated images . The offset is learned to alter the appearance of foreground and background such that a mask can be extracted from the changes in image space . The resulting masks provide segmentation maps for the generated images , but they can not yet be used to segment images from the real world . Given a natural image , the obvious approach would Under review as a conference paper at ICLR 2022 Top images were generated with seeds 5432 and 5431 G en er at ed im ag es an d ex tr ac te d m as ks Re al -w or ld im ag es a nd pr ed ic te d m as ks G en er at ed im ag es an d ex tr ac te d m as ks Re al -w or ld im ag es an d pr ed ic te d m as ks Figure 1 : We automatically find a universal latent direction in a GAN that can separate the foreground from the background without any supervision . We can then generate unlimited samples with masks to train a segmentation network that achieves state-of-the-art unsupervised segmentation performance . be to find the corresponding code in the latent space of the generator , and then obtain a mask with our method . Unfortunately , this inversion process is less than trivial . In fact , recent work provides strong evidence that the expressiveness of GANs is insufficient to encode arbitrary images ( Bau et al. , 2019 ) , meaning that the inversion problem has no solution in general . As we aim to build a general-purpose segmentation method , we take a different approach : we generate a labelled image dataset with foreground/background segmentations and use the generated dataset to train a standard segmentation network . With this , we show that our method can successfully learn accurate foreground-background segmentation networks with no manually provided labels at all . Differently from prior work in GAN-based image segmentation , we neither design a new GAN architecture specifically for the task of segmentation nor use manual supervision to extract segmentation information from an existing GAN . Thus , we can discover meaningful latent directions for any GAN with no need for model-specific manual intervention . Extensive experiments on five segmentation datasets across twelve different GANs demonstrate the effectiveness and generalizability of our approach . Moreover , by constructing our image segmenters from generator networks trained on a generic large-scale datasets such as ImageNet , our method can learn to generically segment objects from a wide range of visual domains . Specifically , when we apply our generic segmenters to the CUB200 ( Welinder et al. , 2010 ) and Oxford Flowers ( Nilsback , 2009 ) datasets , we attain very strong foreground-background results despite not training on this data . Similarly , when we apply our generic segmenters to three saliency detection benchmarks , our method approaches and sometimes even exceeds the performance of supervised and handcrafted saliency detection methods . An analysis of our results also shows that segmentation performance directly correlates with the quality of the underlying GAN , suggesting that foreground/background separation is an important concept in learning generative models . Finally , we demonstrate that our method may be used as a drop-in replacement for saliency networks for the purpose of learning pixel-wise semantic image representations . These pixel-wise representations can then be clustered to obtain fully unsupervised semantic segmentations , extending our method beyond foreground-background segmentation . Thus , we not only demonstrate for the first time that it is possible to obtain semantic segmentations using GANs , but also that this information may be extracted from a wide range of generic GANs trained on general-purpose image datasets . 2 RELATED WORK . Below , we describe how our method relates to recent work in interpreting generative models and object segmentation . Due to space constraints , we include additional related work in the Appendix . Interpreting Deep Generative Models . Several works have proposed methods for decomposing the latent space of a generative model into interpretable or disentangled directions . Early work included Beta-VAE ( Higgins et al. , 2017 ) , which modified the variational ELBO in the original VAE formulation , and InfoGAN ( Chen et al. , 2016 ) , which maximized the mutual information between a subset of the latent code and the generated data . Later work has sought to disentangle factors of variation by mixing latent codes ( Hu et al. , 2018 ) , adding additional adversarial losses ( Mathieu et al. , 2016 ) , and using contrastive learning ( Ren et al. , 2021 ) . Our work follows a recent line of research that looks for structure in large , pretrained generative models . Shen & Zhou ( 2020 ) perform a direct decomposition of model weights to find disentangled directions , while Peebles et al . ( 2020 ) penalize nonzero second-order interactions between different latent dimensions , and Voynov & Babenko ( 2020 ) find interpretable directions by introducing an additional reconstruction network . Differently from the works above , we conduct a deep study of one specific type of structure ( foreground/background separation ) encoded in the latent space . Other works have taken this approach in the context of extracting 3D structure from 2D images ; for example , IG-GAN ( Lunz et al. , 2020 ) uses a neural renderer to recover 3D ( voxel-based ) representations of scenes , and GAN2Shape ( Pan et al. , 2021 ) exploits viewpoint and lighting variations in generated images to recover 3D shapes . Unsupervised Object Segmentation . Prior work on unsupervised object segmentation can be divided into two categories : those that employ generative models to obtain segmentation masks and those that employ purely discriminative methods such as contrastive learning ( Ji et al. , 2019a ; Ouali et al. , 2020 ) . Here , we focus on generative approaches . Nearly all generative approaches are based on the idea of decomposing the generative process in a layer-wise fashion ; in general , the foreground and background of an image are generated separately and then combined to obtain a final image . Specifically , ReDo ( Chen et al. , 2019 ) trains a generator to re-draw new objects on top of old objects , and enforces realism through adversarial training . ( Bielski & Favaro , 2019 ) generates a background , a foreground , and a foreground mask separately and composite them together ; they prevent degenerate outputs ( i.e . the foreground and background being the same ) by randomly shifting the foreground relative to the background . CopyPaste GAN ( Arandjelović & Zisserman , 2019 ) receives two images as input and copies parts of one image onto the other . OneGAN ( Benny & Wolf , 2020 ) learns to simultaneously generate , cluster , and segment images with a combination of GANs , VAEs , and additional encoders . Equivariant Layered GAN ( Yang et al. , 2021 ) first trains a new layerwise GAN and then trains a segmentation network on synthetic data . Labels4Free ( Abdal et al. , 2021 ) proposes a new layerwise network inspired by StyleGAN for foreground-background segmenation . Common difficulties with these layer-wise approaches above include the challenges of training new GANs and scaling beyond simple datasets ( e.g . CUB , Flowers ) . One work along different lines is by Voynov et al . ( 2021 ) , which uses a pretrained BigBiGAN ( Donahue & Simonyan , 2019 ) generator rather than proposing a new layer-wise GAN . Voynov et al . ( 2021 ) use the method from Voynov & Babenko ( 2020 ) to decompose the latent space into interpretable directions . A recent version of their method also picks out this direction without supervision using the idea that a foreground-background separating direction should be decomposable into two affine operators acting on different sets of pixels . Our approach is based on generative modeling , but it differs from other approaches in that we seek a general-purpose method to find foreground/background structure implicitly encoded in standard , non-layer-wise GANs rather than encoding it explicitly or searching for it manually . This enables us to leverage any of the numerous existing generators that have already been pretrained on millions of high-resolution images , rather than developing a new GAN architecture for this specific task . Differently from layer-wise approaches , our approach does not require training new GANs and it does not rely on the assumption that the foreground and background of an image are independent . Differently from Voynov et al . ( 2021 ) , we show through experiments that our method can be applied to a wide range of different GANs and find that it delivers superior performance on four out of five object segmentation and saliency detection benchmarks . Finally , unlike any of these previous works , we demonstrate that our foreground-background segmentation network can be used as a drop-in replacement for saliency networks for the task of learning dense semantic image representations . By clustering these representations , we are able to extend our method from foregound-background segmentation to semantic segmentation . 3 METHOD . Let x ∈ R3×H×W be a ( color ) image . A generator ( network ) is a function G : RD → R3×H×W that maps code variables z to images x = G ( z ) . Optionally , some generative models come with an encoder function E : R3×H×W → RD which computes an approximate inverse of the generator ( i.e . G ( E ( x ) ) ≈ x ) . A challenge in generating images is that individual pixels exhibit complex correlations , caused by the fact that the images are obtained as the composition of a number of different objects . For example , all pixels that belong to a dog have a similar color , characteristic of dog ’ s instance . However , the correlation is much less strong between pixels that belong to different objects . This is because , while object in a scene are not entirely independent , their correlation is much weaker than within the structure of objects . Intuitively , an image generator must learn to account for such correlations in order to generate realistically-looking images . In particular , we expect the generator to capture the idea that pixels that belong to the same object have a related appearance , whereas the appearance of pixels that belong to different objects or , as it may be , to a foreground object and its background , should be much more statistically independent . Given a generator function G , it is then natural to ask whether such correlations can be extracted and used not just for the purpose of generating images , but also for analyzing them . In order to explore this idea , we consider perturbing the code z via a small increment v ∈ RD , where ∈ R and v ∈ SD−1 is a unit vector . Because the dimension D of the embedding space is typically much smaller than the dimension 3HW of the generated images , codes provide highly-compressed views of the data ( for example , D = 120 for BigBiGAN ( Donahue & Simonyan , 2019 ) and the self-conditioned GAN ) . As such , most changes in the code are likely to affect most if not all pixels in the image . However , if the generator did in fact learn to compose objects , then one could hope too find specific variations v that only affect only portions of the image , corresponding to individual objects , and use the latter to highlight and segment them . Empirically , we find that the situation is not as simple . Specifically , it is not easy to find changes in the code that leave part of the pixels exactly constant while changing other pixels . However , we find that there are directions that affect foreground and background regions in a systematic and characteristic manner . Furthermore , we show that these directions are ‘ universal ’ , in the sense that the same v works for all codes z , and are thus characteristic of a given generator network G .

Summary: The paper proposes an unsupervised approach for instance segmentation that leverages pretrained GANs. The approach is divided into two steps: first, the authors discover a latent direction in the generator that allows them to segment generated images. Then, the resulting image-label pairs are used to train a segmentation network. This network can finally be used to perform inference on unseen images. The approach is evaluated on a variety of GAN architectures. The main contribution claimed by the authors is that their method can be applied to off-the-shelf pretrained GANs, whereas competing approaches require architectural variations and re-training.