Self-supervised Training of Proposal-based Segmentation via Background Prediction

While supervised object detection and segmentation methods achieve impressive accuracy , they generalize poorly to images whose appearance significantly differs from the data they have been trained on . To address this in scenarios where annotating data is prohibitively expensive , we introduce a self-supervised approach to detection and segmentation , able to work with monocular images captured with a moving camera . At the heart of our approach lies the observations that object segmentation and background reconstruction are linked tasks , and that , for structured scenes , background regions can be re-synthesized from their surroundings , whereas regions depicting the object can not . We encode this intuition as a self-supervised loss function that we exploit to train a proposal-based segmentation network . To account for the discrete nature of the proposals , we develop a Monte Carlo-based training strategy that allows the algorithm to explore the large space of object proposals . We apply our method to human detection and segmentation in images that visually depart from those of standard benchmarks , achieving competitive results compared to the few existing self-supervised methods and approaching the accuracy of supervised ones that exploit large annotated datasets . 1 INTRODUCTION . Recent object detection and segmentation methods have reached impressive precision and recall rates when trained and tested on large annotated datasets ( Lin et al. , 2014 ) . However , large and varied datasets do not warrant the best possible performance in a particular application domain , as each comes with its own challenges and opportunities . While domain-specific models are thus needed , annotating separate and sufficiently large datasets in all scenarios is impractical . Therefore , weakly- and self-supervised detection and segmentation of salient foreground objects in complex scenes have recently gained attention Croitoru et al . ( 2018 ) ; Eslami et al . ( 2016 ) ; Crawford & Pineau ( 2019 ) ; Rhodin et al . ( 2019 ) ; Bielski & Favaro ( 2019 ) . These methods promise effortless processing of community videos with little human intervention . However , a closer look at existing techniques reveals that they make strong assumptions , such as the target objects being on top of a static background , or rely on pre-trained supervised object localization , object-boundary detection , and optical flow networks . This severely limits their applicability in practice . To develop a more generic technique , we start from the observation that in most images the background forms a consistent , natural scene . Therefore the appearance of any background patch can be predicted from its surroundings . By contrast , a salient object ’ s appearance is unpredictable from the neighboring scene content and can be expected to be very different from what an inpainting algorithm would produce . We incorporate this insight into a proposal-generating deep network whose architecture is inspired by those of YOLO Redmon et al . ( 2016 ) and MaskRCNN He et al . ( 2017 ) but does not require explicit supervision . For each proposal , we synthesize a background image by masking out the corresponding region and inpainting it from the remaining image . The loss function we minimize favors the largest possible distance between this reconstructed background and the input image . This encourages the network to select regions that can not be explained from their surrounding and are therefore salient . To handle the discrete nature of the proposals , we introduce a Monte Carlo-based strategy to train our network . It operates on a discrete distribution , is unbiased , exhibits low variance , and is end-to-end trainable . We tailor our training algorithm to segmenting human motion from videos in which a single subject moves . The constraints imposed by this well-defined , yet challenging and widely studied task , allow us to generalize to arbitrary , dynamic camera motion . As a consequence , it lets us perform end-to-end training without the bootstrapping Croitoru et al . ( 2019 ) and intermediate optical flow supervision Papazoglou & Ferrari ( 2013 ) ; Jain et al . ( 2017 ) ; Tokmakov et al . ( 2017 ) . We demonstrate the effectiveness of our unsupervised method on several datasets captured with increasingly mobile cameras , ranging from static to pan-tilt-zoom and hand-held . We will show that our approach applies to images acquired in conditions significantly more general than those of standard benchmarks , without requiring any manual annotation . Thus , as shown in Fig . 1 , it approaches the quality and sometimes outperforms state-of-the-art detectors that have been trained on large annotated datasets . Retraining or fine-tuning these methods on this data could be done but would require supervision that is hard to obtain , which makes a self-supervised approach attractive . We will make our code and ground-truth segmentations for the existing Ski-PTZ-Dataset and the new Handheld190k dataset publicly available upon acceptance of the paper . 2 RELATED WORK . Most salient object detection and segmentation algorithms are fully-supervised ( Redmon et al. , 2016 ; He et al. , 2017 ; Song et al. , 2018 ; Cheng et al. , 2017 ) and require large annotated datasets with paired images and labels . Our goal is a purely self-supervised method that succeeds without segmentation and object bounding box annotations . Note that this differs from the so-called unsupervised object detection methods ( Hu et al. , 2018 ; Li et al. , 2018 ; Jain et al. , 2017 ) , which still require domain-specific annotations at training but not at test time . We focus our discussion on self- and weakly-supervised methods with regard to the type of training data used and refer to ( Koh & Kim , 2017 ) for the discussion of methods using hand-crafted optimization . Weakly-supervised methods . A classical weakly-supervised example is the Hough Matching algorithm ( Cho et al. , 2015 ) . It uses an object classification dataset and identifies foreground as the image regions that have re-occurring Hough features within images of the same class . Similar principles have been followed using deep networks trained for object detection ( Wei et al. , 2017 ; Jain et al. , 2017 ) , optical flow ( Tokmakov et al. , 2017 ) , and object saliency ( Li et al. , 2018 ) . These methods make the implicit assumption that the background varies across examples and can therefore be excluded as noise . This assumption is violated in the targeted case of training on domain-specific images , where foreground and background are similar across examples . Motion-based methods Given video sequences , the temporal information can be exploited by assuming that the background changes slowly ( Barnich & Van Droogenbroeck , 2011 ) and linearly ( Stretcu & Leordeanu , 2015 ) . However , even a static scene induces non-homogeneous deformations under camera translation , and it can be difficult to handle all types of camera motion ( pan , tilt , zoom ) within a single video , and distinguish articulated human motion from background motion ( Russell et al. , 2014 ) . Some of the resulting errors can be corrected by iteratively refining the crude background subtraction results of Stretcu & Leordeanu ( 2015 ) with an ensemble of student and teacher networks ( Croitoru et al. , 2018 ; 2019 ) . This , however , induces a strong dependence on the teacher used for bootstrapping . Our approach is conceptually closely related to VideoPCA ( Stretcu & Leordeanu , 2015 ) , which models the background as the part of the scene that can be explained by a low-dimensional linear basis . This implicitly assumes that the foreground is harder to model than the background and can therefore be separated as the non-linear residual . Here , instead of using motion cues , we propose to rely on the predictability of image patches from their spatial neighborhood . Self-supervised Methods . Most similar to our approach are the self-supervised ones to objectdetection ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Rhodin et al. , 2019 ; Bielski & Favaro , 2019 ) that complement auto-encoder networks by an attention mechanism . These methods first detect one or several bounding boxes whose content is extracted using a spatial transformer ( Jaderberg et al. , 2015 ) . This content is then passed through an auto-encoder and re-composited with a background . In ( Rhodin et al. , 2019 ) , the background is assumed to be static and in ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ) even single colored , a severe restriction in practice . Crawford & Pineau ( 2019 ) use a proposal-based network similar to ours , but resort to approximating the proposal distribution with a continuous one to make the model differentiable . Here , we demonstrate that much simpler importance sampling is sufficient . In ( Bielski & Favaro , 2019 ) , a generative model relies on the assumption that the image region strictly covering the salient object can be subject to random shifts without affecting the realism of the scene . However , this method can easily be deceived by other background objects whose random displacement can still yield realistic images . By contrast to all of these methods , our approach works with images acquired using a moving camera and given an arbitrarily colored background . In addition to object detection , the algorithm of Rhodin et al . ( 2019 ) also returns instance segmentation masks by reasoning about the extent and depth ordering of multiple people in a multi-camera scene . However , this requires multiple static cameras and a static background at training time , as does the approach of Baqué et al . ( 2017 ) that performs instance segmentation in crowded scenes . 3 METHOD . Our goal is to learn a salient person detector and segmentor from unlabeled videos acquired in as generic a setup as possible , including using hand-held cameras . At inference time , our algorithm takes a single image I ∈ RW×H as input and outputs a bounding box bm ∈ R4 , expressed as a center location , width , and height , and a segmentation mask S ∈ R128×128 within that window . This is achieved by the multi-stage process visualized in Fig . 2 . A first network predicts a set of C candidate object locations ( bc ) Cc=1 and corresponding probabilities ( pc ) C c=1 . To this end , we use a fully-convolutional architecture that divides the image into a grid , with each cell c yielding one probability , pc , and an offset from the cell center to compute bc , similarly to YOLO Redmon et al . ( 2016 ) . Second , the bm with highest probability pm ≥ pc , ∀c , is chosen and its content is decoded into foreground Î ∈ R128×128 , segmentation mask S , and background B ∈ RW×H with separate encoder-decoder branches . 3.1 SELF-SUPERVISED TRAINING WITH KNOWN BACKGROUND . Given a set of unlabeled images ( Ii ) Ni=1 , our goal is to train a neural networkD ( I ) 7→ ( ( bc ) , ( pc ) ) Cc=1 to propose suitable bounding box candidates and the corresponding probabilities to select the bm that contains an object . Because object locations are unknown , we do this with an autoencoder objective , L ( F ( I , bc , B ) , I ) , that measures how well the autoencoder F reproduces the input image I on top of a background B , with the attention on bc . As in ( Crawford & Pineau , 2019 ; Rhodin et al. , 2019 ) , this is implemented with a spatial transformer T that crops the area of interest , followed by a bottle-neck autoencoder S that produces the foreground and segmentation mask , and a second spatial transformer T −1 that undoes the cropping and blends the synthesized foreground and background . We write F ( I , bc , B ) = T −1 ( Î ) ◦ T −1 ( S ) +B ◦ ( 1− ( T −1 ( S ) ) , with S ( T ( I , b ) ) 7→ ( Î , S ) , ( 1 ) where ◦ is the elementwise multiplication . The loss L can be a least-square distance between all pixel values , alternatives are discussed at the end of the section . Because the attention window b selects part of the image for decoding , this loss encourages D to focus on the object so as to model the foreground on top of B . Moreover , the autoencoder only approximates the actual image , forcing the segmentation mask to contain only the parts not captured by the background . For now , we assume B to be known , but we will remove this assumption later . In this setting , we derive a probabilistic formulation in which not a single , but multiple candidates can give rise to plausible reconstructions . We then reason about the expected loss across all likely candidates , O ( I ) = Ec [ L ( F ( I , bc , B ) , I ) ] , with c ∼ p , where p ( c ) = pc , ( 2 ) where Ec denotes the expectation over c drawn from the categorical proposal distribution output by the network D ( I ) . We optimize Eq . ( 2 ) with stochastic gradient descent and mini-batches of ( Ii ) Ni=1 . Note that minimizing this objective will jointly optimize the detection network D that generates and stochastically selects proposals , and the synthesis network S that models the object appearance and the segmentation mask . Because we have a finite set of candidates , Eq . ( 2 ) can be expressed deterministically as a weighted sum over all candidates as O ( I ) = C∑ c=1 p ( c ) L ( F ( I , bc , B ) , I ) . ( 3 ) This objective is an explicit function of p and could be optimized by gradient descent . However , this sum over C forward passes is inefficient to evaluate in practice when C is large . For example , in our experiments C = 64 , which does not fit in memory . This was also observed by Crawford & Pineau ( 2019 ) , who resorted to using a continuous approximation of the discrete distribution p to facilitate end-to-end training . Here , we propose a simpler alternative , exploiting Monte Carlo and importance sampling , which provides an unbiased estimator with low variance . Monte Carlo ( MC ) sampling . In principle , the expectation in Eq . ( 2 ) could be estimated by sampling a small set of J candidate cells from p. Unfortunately , sampling from such a discrete distribution is not differentiable with respect to its parameters , which precludes end-to-end gradientbased optimization . Instead of sampling according to p , we can rewrite Eq . ( 2 ) using an arbitrary distribution q , by reweighting with the quotient of both distributions . That is , O ( I ) = Ec [ p ( c ) q ( c ) L ( F ( I , bc , B ) , I ) ] , with c ∼ q . ( 4 ) This change of distribution and relative weighting holds for any two probability distributions , as explained in the appendix . In practice , we approximate the expectation on a mini-batch , with a single sample drawn from q per image . Importance sampling . While moving the distribution into the expectation sum provides differentiability , it comes with the drawback of a potentially large variance , i.e. , high approximation error for few samples . For instance , by choosing q to be the uniform sampling distribution U , most of the uniformly drawn samples will have a low probability in p and therefore negligible influence . To reduce this variance , we leverage importance sampling and set the sampling distribution q to be similar or equivalent to p. To prevent division by very small values that could lead to numerical instability , we define the new distribution q as q ( c ) = p ( c ) ( 1− C ) + . ( 5 ) As a side effect , controls the probability that an unlikely case is chosen , which induces a form of exploration that is helpful in the early training stages of the network . Note that the fraction p ( c ) q ( c ) cancels numerically for q ( c ) = p ( c ) . However , while values cancel , their derivatives do not ; the differentiability of p is maintained as the sampling distribution q must be treated as a constant . In this case , the gradient of Eq . 4 equals that of the likelihood ratio method ( Glynn , 1990 ) used in the REINFORCE algorithm ( Williams , 1992 ) . We favor the MC interpretation that explains the effect of the gradient quotient as importance sampling and generalizes to arbitrary q . While the relation is known for a long time ( Glynn , 1990 ) , is overlooked in the recent literature .

Summary: This paper provides a new self-supervised proposal-based approach for object detection and segmentation. The author introduces a Monte Carlo-based optimization to solve the inefficiency problem in the discrete proposal-based forward process defined in (Crawford and Pineau 2019). Also, the paper redefines the decoder part for self-supervised from minimizing reconstruction loss with background segmentation to maximize reconstruction error with learning a foreground segmentation.  The method is then verified with a suite of experiments for people-detection on video datasets.