Unsupervised Foreground Extraction via Deep Region Competition

1 Introduction . Foreground extraction , being a special case of generic image segmentation , aims for a binary partition of the given image with specific semantic meaning , i.e. , a foreground that typically contains identifiable objects and the possibly less structural remaining regions as the background . There is a rich literature on explicitly modeling and representing a given image as foreground and background ( or more general visual regions ) , such that a generic inference algorithm can produce plausible segmentations ideally for any images without or with little supervision [ 1–8 ] . However , such methods essentially rely on low-level visual features ( e.g. , edges , color , and texture ) , and some further require human intervention at initialization [ 4 , 5 ] , which largely limits their practical performance on modern datasets of complex natural images with rich semantic meanings [ 9 , 10 ] . These datasets typically come with fine-grained semantic annotations , exploited by supervised methods that learn representation and inference algorithm as one monolithic network [ 11–16 ] . Despite the success of densely supervised learning , the unsupervised counterpart is still favored due to its resemblance to how humans perceive the world [ 17 , 18 ] . 1Code and data available at https : //github.com/yuPeiyu98/DRC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Attempting to combine unsupervised or weakly supervised learning with modern neural networks , three lines of work surge recently for foreground extraction : ( 1 ) deep networks as feature extractors for canonical segmentation algorithms , ( 2 ) GAN-based foreground-background disentanglement , and ( 3 ) compositional latent variable models with slot-based object modeling . Despite great successes of these methods , the challenge of unsupervised foreground extraction remains largely open . Specifically , the first line of work trains designated deep feature extractors for canonical segmentation algorithms or metric networks as learned partitioning criteria [ 19–21 ] . These methods ( e.g. , W-Net [ 19 ] ) define foreground objects ’ properties using learned features or criteria and are thus generally bottle-necked by the selected post-processing segmentation algorithm [ 22 , 23 ] . As a branch of pioneering work that moves beyond these limitations , Yang et al . [ 24 , 25 ] have recently proposed a general contextual information separation principle and an efficient adversarial learning method that is generally applicable to unsupervised segmentation , separation and detection . GAN-based models [ 26–31 ] capture the foreground objectness with oversimplified assumptions or require additional supervision to achieve foreground-background disentanglement . For example , the segmentation model in ReDO [ 28 ] is trained by redrawing detected objects , which potentially limits its application to datasets with diverse object shapes . OneGAN [ 31 ] and its predecessors [ 29 , 30 ] , though producing impressive results on foreground extraction , require a set of background images without foreground objects as additional inputs . Lastly , compositional latent variable models [ 32–40 ] include the background as a “ virtual object ” and induce the independence of object representations using an identical generator for all object slots . Although these methods exhibit strong performance on synthetic multi-object datasets with simple backgrounds and foreground shapes , they may fail on complex real-world data or even synthetic datasets with more challenging backgrounds [ 37 , 38 ] . In addition , few unsupervised learning methods have provided explicit identification of foreground objects and background regions . While they can generate valid segmentation masks , most of these methods do not specify which output corresponds to the foreground objects . These deficiencies necessitate rethinking the problem of unsupervised foreground extraction . We propose to confront the challenges in formulating ( 1 ) a generic inductive bias for modeling foreground and background regions that can be baked into neural generators , and ( 2 ) an effective inference algorithm based on a principled criterion for foreground-background partition . Inspired by Region Competition [ 1 ] , a seminal approach that combines optimization-based inference [ 41–43 ] and probabilistic visual modeling [ 44 , 45 ] by minimizing a generalized Bayes criterion [ 46 ] , we propose to solve the foreground extraction problem by reconciling energy-based prior [ 47 ] with generative image modeling in the form of Mixture of Experts ( MoE ) [ 48 , 49 ] . To generically describe background regions , we further introduce the learned pixel re-assignment as the essential inductive bias to capture their regularities . Fueled by our modeling , we propose to find the foreground-background partition through Expectation-Maximization ( EM ) . Our algorithm effectively exploits the interaction between the mixture components during the partitioning process , echoing the intuition described in Region Competition [ 1 ] . We therefore coin our method Deep Region Competition ( DRC ) . We summarize our contributions as follows : 1 . We provide probabilistic foreground-background modeling by reconciling energy-based prior with generative image modeling in the form of MoE . With this modeling , the foregroundbackground partition can be naturally produced through EM . We further introduce an inductive bias , pixel re-assignment , to facilitate foreground-background disentanglement . 2 . In experiments , we demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods . Furthermore , we empirically show that using learned pixel re-assignment as the inductive bias helps to provide explicit identification for foreground and background regions . 3 . We find that DRC can potentially generalize to novel foreground objects even from categories unseen during training , which may provide some inspiration for the study of out-of-distribution ( OOD ) generalization in more general unsupervised disentanglement . 2 Related Work . A typical line of methods frames unsupervised or weakly supervised foreground segmentation within a generative modeling context . Several methods build upon generative adversarial networks ( GAN ) [ 26 ] to perform foreground segmentation . LR-GAN [ 27 ] learns to generate background re- gions and foreground objects separately and recursively , which simultaneously produces the foreground objects mask . ReDO ( ReDrawing of Objects ) [ 28 ] proposes a GAN-based object segmentation model , based on the assumption that replacing the foreground object in the image with a generated one does not alter the distribution of the training data , given that the foreground object is correctly discovered . Similarly , SEIGAN [ 29 ] learns to extract foreground objects by recombining the foreground objects with the generated background regions . FineGAN [ 30 ] hierarchically generates images ( i.e. , first specifying the object shape and then the object texture ) to disentangle the background and foreground object . Benny and Wolf [ 31 ] further hypothesize that a method solving an ensemble of unsupervised tasks altogether improves the model performance compared with the one that solves each individually . Therefore , they train a complex GAN-based model ( OneGAN ) to solve several tasks simultaneously , including foreground segmentation . Although LR-GAN and FineGAN do produce masks as part of their generative process , they can not segment a given image . Despite SEIGAN and OneGAN achieving decent performance on foreground-background segmentation , these methods require a set of clean background images as additional inputs for weak supervision . ReDO captures the foreground objectness with possibly oversimplified assumptions , limiting its application to datasets with diverse object shapes . On another front , compositional generative scene models [ 32–40 ] , sharing the idea of scene decomposition stemming from DRAW [ 50 ] , learn to represent foreground objects and background regions in terms of a collection of latent variables with the same representational format . These methods typically exploit the spatial mixture model for generative modeling . Specifically , IODINE [ 37 ] proposes a slot-based object representation method and models the latent space using iterative amortized inference [ 51 ] . Slot-Attention [ 38 ] , as a step forward , effectively incorporates the attention mechanism into the slot-based object representation for flexible foreground object binding . Both methods use fully shared parameters among individual mixture components to entail permutation invariance of the learned multi-object representation . Alternative models such as MONet [ 36 ] and GENESIS [ 39 ] use multiple encode-decode steps for scene decomposition and foreground object extraction . Although these methods exhibit strong performance on synthetic multi-object datasets with simple background and foreground shapes , they may fail when dealing with complex real-world data or even synthetic datasets with more challenging background [ 37 , 38 ] . More closely related to the classical methods , another line of work focuses on utilizing image features extracted by deep neural networks or designing energy functions based on data-driven methods to define the desired property of foreground objects . Pham et al . [ 52 ] and Silberman et al . [ 53 ] obtain impressive results when depth images are accessible in addition to conventional RGB images , while such methods are not directly applicable for data with RGB images alone . W-Net [ 19 ] extracts image features via a deep auto-encoder jointly trained by minimizing reconstruction error and normalized cut . The learned features are further processed by CRF smoothing to perform hierarchical segmentation . Kanezaki [ 20 ] proposes to employ a neural network as part of the partitioning criterion ( inspired by Ulyanov et al . [ 54 ] ) to minimize the chosen intra-region pixel distance for segmentation directly . Ji et al . [ 21 ] propose to use Invariant Information Clustering as the objective for segmentation , where the network is trained to be part of the learned distance . As an interesting extension , one may also consider adapting methods that automatically discover object structures [ 55 ] to foreground extraction . Though being pioneering work in image segmentation , the aforementioned methods are generally bottle-necked by the selected post-processing segmentation algorithm or require extra transformations to produce meaningful foreground segmentation masks . Yang et al . [ 24 , 25 ] in their seminal work propose an information-theoretical principle and adversarial contextual model for unsupervised segmentation and detection by partitioning images into maximally independent sets , with the objective of minimizing the predictability of one set by the other sets . Additional efforts have also been devoted to weakly supervised foreground segmentation using image classification labels [ 56–58 ] , bounding boxes [ 59 , 60 ] , or saliency maps [ 61–63 ] . 3 Methodology . Foreground extraction performs a binary partition for the image I to extract the foreground region . Without explicit supervision , we propose to use learned pixel re-assignment as a generic inductive bias for background modeling , upon which we derive an EM-like partitioning algorithm . Compared with prior methods , our algorithm can handle images with more complex foreground shapes and background patterns , while providing explicit identification of foreground and background regions . 3.1 Preliminaries . Adopting the language of EM algorithm , we assume that for the observed sample x P RD , there exists z P Rd as its latent variables . The complete-data distribution is pθpz , xq “ pαpzqpβpx|zq , ( 1 ) where pαpzq is the prior model with parameters α , pβpx|zq is the top-down generative model with parameters β , and θ “ pα , βq . The prior model pαpzq can be formulated as an energy-based model , which we refer to as the Latentspace Energy-Based Model ( LEBM ) [ 47 ] throughout the paper : pαpzq “ 1 Zα exp pfαpzqq p0pzq , ( 2 ) where fαpzq can be parameterized by a neural network , Zα is the partition function , and p0pzq is a reference distribution , assumed to be isotropic Gaussian prior commonly used for the generative model . The prior model in Eq . ( 2 ) can be interpreted as an energy-based correction or exponential tilting of the original prior distribution p0 . The LEBM can be learned by Maximum Likelihood Estimation ( MLE ) . Given a training sample x , the learning gradient for α is derived as shown by Pang et al . [ 47 ] , δαpxq “ Epθpz|xq r∇αfαpzqs ´ Epαpzq r∇αfαpzqs . ( 3 ) In practice , the above expectations can be approximated by Monte-Carlo average , which requires sampling from pθpz|xq and pαpzq . This step can be done with stochastic gradient-based methods , such as Langevin dynamics [ 64 ] or Hamiltonian Monte Carlo [ 65 ] . An extension to LEBM is to further couple the vector representation z with a symbolic representation y [ 66 ] . Formally , y is a K-dimensional one-hot vector , where K is the number of possible z categories . Such symbol-vector duality can provide extra entries for auxiliary supervision ; we will detail it in Section 3.4 .

Summary: In this work, the authors propose an unsupervised, generative model for foreground-background segmentation of RGB images which is based on the previously proposed "Latent Space Engergy-Based Prior Model" (LEBM). Similar to a VAE, a latent vector sampled from a prior distribution is transformed into an image by a decoder network. Differently from VAEs however, the LEBM uses a more complex, learnable prior distribution and sampling from the prior and posterior distributions is realized by differentiable MCMC sampling using Langevin dynamics. An image is modelled using a spatial mixture of foreground and background images decoded by separate decoder networks. To foster the segregation into foreground and background, the backround model includes an additional, spatial resampling step targeted at the assumed background statistics. While the reconstructed images tend to be blurry, the reconstructed foreground-background masks have been shown to be more accurate than previous methods on several datasets.