Unsupervised Foreground Extraction via Deep Region Competition

1 Introduction . Foreground extraction , being a special case of generic image segmentation , aims for a binary partition of the given image with specific semantic meaning , i.e. , a foreground that typically contains identifiable objects and the possibly less structural remaining regions as the background . There is a rich literature on explicitly modeling and representing a given image as foreground and background ( or more general visual regions ) , such that a generic inference algorithm can produce plausible segmentations ideally for any images without or with little supervision [ 1–8 ] . However , such methods essentially rely on low-level visual features ( e.g. , edges , color , and texture ) , and some further require human intervention at initialization [ 4 , 5 ] , which largely limits their practical performance on modern datasets of complex natural images with rich semantic meanings [ 9 , 10 ] . These datasets typically come with fine-grained semantic annotations , exploited by supervised methods that learn representation and inference algorithm as one monolithic network [ 11–16 ] . Despite the success of densely supervised learning , the unsupervised counterpart is still favored due to its resemblance to how humans perceive the world [ 17 , 18 ] . 1Code and data available at https : //github.com/yuPeiyu98/DRC 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Attempting to combine unsupervised or weakly supervised learning with modern neural networks , three lines of work surge recently for foreground extraction : ( 1 ) deep networks as feature extractors for canonical segmentation algorithms , ( 2 ) GAN-based foreground-background disentanglement , and ( 3 ) compositional latent variable models with slot-based object modeling . Despite great successes of these methods , the challenge of unsupervised foreground extraction remains largely open . Specifically , the first line of work trains designated deep feature extractors for canonical segmentation algorithms or metric networks as learned partitioning criteria [ 19–21 ] . These methods ( e.g. , W-Net [ 19 ] ) define foreground objects ’ properties using learned features or criteria and are thus generally bottle-necked by the selected post-processing segmentation algorithm [ 22 , 23 ] . As a branch of pioneering work that moves beyond these limitations , Yang et al . [ 24 , 25 ] have recently proposed a general contextual information separation principle and an efficient adversarial learning method that is generally applicable to unsupervised segmentation , separation and detection . GAN-based models [ 26–31 ] capture the foreground objectness with oversimplified assumptions or require additional supervision to achieve foreground-background disentanglement . For example , the segmentation model in ReDO [ 28 ] is trained by redrawing detected objects , which potentially limits its application to datasets with diverse object shapes . OneGAN [ 31 ] and its predecessors [ 29 , 30 ] , though producing impressive results on foreground extraction , require a set of background images without foreground objects as additional inputs . Lastly , compositional latent variable models [ 32–40 ] include the background as a “ virtual object ” and induce the independence of object representations using an identical generator for all object slots . Although these methods exhibit strong performance on synthetic multi-object datasets with simple backgrounds and foreground shapes , they may fail on complex real-world data or even synthetic datasets with more challenging backgrounds [ 37 , 38 ] . In addition , few unsupervised learning methods have provided explicit identification of foreground objects and background regions . While they can generate valid segmentation masks , most of these methods do not specify which output corresponds to the foreground objects . These deficiencies necessitate rethinking the problem of unsupervised foreground extraction . We propose to confront the challenges in formulating ( 1 ) a generic inductive bias for modeling foreground and background regions that can be baked into neural generators , and ( 2 ) an effective inference algorithm based on a principled criterion for foreground-background partition . Inspired by Region Competition [ 1 ] , a seminal approach that combines optimization-based inference [ 41–43 ] and probabilistic visual modeling [ 44 , 45 ] by minimizing a generalized Bayes criterion [ 46 ] , we propose to solve the foreground extraction problem by reconciling energy-based prior [ 47 ] with generative image modeling in the form of Mixture of Experts ( MoE ) [ 48 , 49 ] . To generically describe background regions , we further introduce the learned pixel re-assignment as the essential inductive bias to capture their regularities . Fueled by our modeling , we propose to find the foreground-background partition through Expectation-Maximization ( EM ) . Our algorithm effectively exploits the interaction between the mixture components during the partitioning process , echoing the intuition described in Region Competition [ 1 ] . We therefore coin our method Deep Region Competition ( DRC ) . We summarize our contributions as follows : 1 . We provide probabilistic foreground-background modeling by reconciling energy-based prior with generative image modeling in the form of MoE . With this modeling , the foregroundbackground partition can be naturally produced through EM . We further introduce an inductive bias , pixel re-assignment , to facilitate foreground-background disentanglement . 2 . In experiments , we demonstrate that DRC exhibits more competitive performances on complex real-world data and challenging multi-object scenes compared with prior methods . Furthermore , we empirically show that using learned pixel re-assignment as the inductive bias helps to provide explicit identification for foreground and background regions . 3 . We find that DRC can potentially generalize to novel foreground objects even from categories unseen during training , which may provide some inspiration for the study of out-of-distribution ( OOD ) generalization in more general unsupervised disentanglement . 2 Related Work . A typical line of methods frames unsupervised or weakly supervised foreground segmentation within a generative modeling context . Several methods build upon generative adversarial networks ( GAN ) [ 26 ] to perform foreground segmentation . LR-GAN [ 27 ] learns to generate background re- gions and foreground objects separately and recursively , which simultaneously produces the foreground objects mask . ReDO ( ReDrawing of Objects ) [ 28 ] proposes a GAN-based object segmentation model , based on the assumption that replacing the foreground object in the image with a generated one does not alter the distribution of the training data , given that the foreground object is correctly discovered . Similarly , SEIGAN [ 29 ] learns to extract foreground objects by recombining the foreground objects with the generated background regions . FineGAN [ 30 ] hierarchically generates images ( i.e. , first specifying the object shape and then the object texture ) to disentangle the background and foreground object . Benny and Wolf [ 31 ] further hypothesize that a method solving an ensemble of unsupervised tasks altogether improves the model performance compared with the one that solves each individually . Therefore , they train a complex GAN-based model ( OneGAN ) to solve several tasks simultaneously , including foreground segmentation . Although LR-GAN and FineGAN do produce masks as part of their generative process , they can not segment a given image . Despite SEIGAN and OneGAN achieving decent performance on foreground-background segmentation , these methods require a set of clean background images as additional inputs for weak supervision . ReDO captures the foreground objectness with possibly oversimplified assumptions , limiting its application to datasets with diverse object shapes . On another front , compositional generative scene models [ 32–40 ] , sharing the idea of scene decomposition stemming from DRAW [ 50 ] , learn to represent foreground objects and background regions in terms of a collection of latent variables with the same representational format . These methods typically exploit the spatial mixture model for generative modeling . Specifically , IODINE [ 37 ] proposes a slot-based object representation method and models the latent space using iterative amortized inference [ 51 ] . Slot-Attention [ 38 ] , as a step forward , effectively incorporates the attention mechanism into the slot-based object representation for flexible foreground object binding . Both methods use fully shared parameters among individual mixture components to entail permutation invariance of the learned multi-object representation . Alternative models such as MONet [ 36 ] and GENESIS [ 39 ] use multiple encode-decode steps for scene decomposition and foreground object extraction . Although these methods exhibit strong performance on synthetic multi-object datasets with simple background and foreground shapes , they may fail when dealing with complex real-world data or even synthetic datasets with more challenging background [ 37 , 38 ] . More closely related to the classical methods , another line of work focuses on utilizing image features extracted by deep neural networks or designing energy functions based on data-driven methods to define the desired property of foreground objects . Pham et al . [ 52 ] and Silberman et al . [ 53 ] obtain impressive results when depth images are accessible in addition to conventional RGB images , while such methods are not directly applicable for data with RGB images alone . W-Net [ 19 ] extracts image features via a deep auto-encoder jointly trained by minimizing reconstruction error and normalized cut . The learned features are further processed by CRF smoothing to perform hierarchical segmentation . Kanezaki [ 20 ] proposes to employ a neural network as part of the partitioning criterion ( inspired by Ulyanov et al . [ 54 ] ) to minimize the chosen intra-region pixel distance for segmentation directly . Ji et al . [ 21 ] propose to use Invariant Information Clustering as the objective for segmentation , where the network is trained to be part of the learned distance . As an interesting extension , one may also consider adapting methods that automatically discover object structures [ 55 ] to foreground extraction . Though being pioneering work in image segmentation , the aforementioned methods are generally bottle-necked by the selected post-processing segmentation algorithm or require extra transformations to produce meaningful foreground segmentation masks . Yang et al . [ 24 , 25 ] in their seminal work propose an information-theoretical principle and adversarial contextual model for unsupervised segmentation and detection by partitioning images into maximally independent sets , with the objective of minimizing the predictability of one set by the other sets . Additional efforts have also been devoted to weakly supervised foreground segmentation using image classification labels [ 56–58 ] , bounding boxes [ 59 , 60 ] , or saliency maps [ 61–63 ] . 3 Methodology . Foreground extraction performs a binary partition for the image I to extract the foreground region . Without explicit supervision , we propose to use learned pixel re-assignment as a generic inductive bias for background modeling , upon which we derive an EM-like partitioning algorithm . Compared with prior methods , our algorithm can handle images with more complex foreground shapes and background patterns , while providing explicit identification of foreground and background regions . 3.1 Preliminaries . Adopting the language of EM algorithm , we assume that for the observed sample x P RD , there exists z P Rd as its latent variables . The complete-data distribution is pθpz , xq “ pαpzqpβpx|zq , ( 1 ) where pαpzq is the prior model with parameters α , pβpx|zq is the top-down generative model with parameters β , and θ “ pα , βq . The prior model pαpzq can be formulated as an energy-based model , which we refer to as the Latentspace Energy-Based Model ( LEBM ) [ 47 ] throughout the paper : pαpzq “ 1 Zα exp pfαpzqq p0pzq , ( 2 ) where fαpzq can be parameterized by a neural network , Zα is the partition function , and p0pzq is a reference distribution , assumed to be isotropic Gaussian prior commonly used for the generative model . The prior model in Eq . ( 2 ) can be interpreted as an energy-based correction or exponential tilting of the original prior distribution p0 . The LEBM can be learned by Maximum Likelihood Estimation ( MLE ) . Given a training sample x , the learning gradient for α is derived as shown by Pang et al . [ 47 ] , δαpxq “ Epθpz|xq r∇αfαpzqs ´ Epαpzq r∇αfαpzqs . ( 3 ) In practice , the above expectations can be approximated by Monte-Carlo average , which requires sampling from pθpz|xq and pαpzq . This step can be done with stochastic gradient-based methods , such as Langevin dynamics [ 64 ] or Hamiltonian Monte Carlo [ 65 ] . An extension to LEBM is to further couple the vector representation z with a symbolic representation y [ 66 ] . Formally , y is a K-dimensional one-hot vector , where K is the number of possible z categories . Such symbol-vector duality can provide extra entries for auxiliary supervision ; we will detail it in Section 3.4 .

Summary: This work proposes a novel unsupervised foreground segmentation technique based on Mixture of Experts energy based modelling with a generative neural network. They introduce an interesting inductive bias, pixel reassignment, which works as the main regularizer and the secret sauce that enables the fg/bg separation. They essentially reshuffle the pixels for the background generation based on the learned index matrix and ablations shows that's the most significant regularizer.  They compare their results on 6 datasets, including birds, dogs, and clevr6 to both object centric approaches such as slot attention and gan based methods. They consistently outperform the previous methods with a significant margin.