Self-supervised Training of Proposal-based Segmentation via Background Prediction
Authors: Isinsu Katircioglu
Helge Rhodin
Victor Constantin
Jörg Spörri
Mathieu Salzmann
Pascal Fua
Venue:  (2020)

Abstract: While supervised object detection and segmentation methods achieve impressive accuracy, they generalize poorly to images whose appearance significantly differs from the data they have been trained on. To address this in scenarios where annotating data is prohibitively expensive, we introduce a self-supervised approach to detection and segmentation, able to work with monocular images captured with a moving camera. At the heart of our approach lies the observations that object segmentation and background reconstruction are linked tasks, and that, for structured scenes, background regions can be re-synthesized from their surroundings, whereas regions depicting the object cannot.
We encode this intuition as a self-supervised loss function that we exploit to train a proposal-based segmentation network. To account for the discrete nature of the proposals, we develop a Monte Carlo-based training strategy that allows the algorithm to explore the large space of object proposals. We apply our method to human detection and segmentation in images that visually depart from those of standard benchmarks, achieving competitive results compared to the few existing self-supervised methods and approaching the accuracy of supervised ones that exploit large annotated datasets.

Under review as a conference paper at ICLR 2020
SELF-SUPERVISED TRAINING OF PROPOSAL-BASED SEGMENTATION VIA BACKGROUND PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
While supervised object detection and segmentation methods achieve impressive accuracy, they generalize poorly images whose appearance significantly differs from the data they have been trained on. To address this in scenarios where anno- tating data prohibitively expensive, we introduce self-supervised approach detection and segmentation, able to work with monocular images captured with a moving camera. At the heart of our approach lies the observations that object seg- mentation and background reconstruction are linked tasks, and that, for structured scenes, background regions can be re-synthesized from their surroundings, whereas regions depicting the object cannot. We encode this intuition as self-supervised loss function that we exploit to train a proposal-bas bbased segmentation network. To account for the discrete nature of the proposals, we develop Monte Carlo-based training strategy that allows the algorithm to explore the large space of object proposals. We apply method to human detection and segmentation in images that visually depart from those standard benchmarks, achieving competitive results compared to the few existing self- supe-sueeeeieed methods and approaching the accuracy of supervised ones that exploit large annotated datasets.
INTRODUCTION
Recent object detection and segmentation methods have reached impressive precision and recall rates when trained and tested on large annotated datasets (Lin et al. 2014). However, large and varied particular application domain, as each datasets do not warrant the best possible performance comes with its own challenges and opportunities. While domain-specifie models thus needed, annotating separate and sufficiently large datasets in all scenarios is impractical.
Therefore, weakly- and self-supervised detection and segmentation of salient foreground objects in complex scenes have recently gained attention Croitoru et al. (2018); Eslami et al. 2016); Crawford & Pineau (2019); Rhodin et al. (2019); Bielski & Favaro (2019). These methods promise effortless processing of community videos with little human intervention. However, a closer look at existing techniques reveals that they make strong assumptions, such the target objects being top of static background. or rely on pre- pre-trained supervised object localization, sodecttbooudary detection, and optical flow networks. This severely limits their applicability in practice.
To develop more generic technique, we start from the observation that in most images the background forms consistent, natural scene. Therefore the appearance of any background patch can be predicted from its surroundings. By contrast, salient object's 's appearance is unpredictable from the neighboring scene content and can be expected be very different from what an inpainting algorithm would produce. We incorporate this insight into proposal-generati inspired by those of YOLO Redmon et al. (2016) and MaskRCNN He et al. (2077) but does not require explicit supervision.
deep network whose architecture
For each proposal, we synthesize a background image by masking out the corresponding region and inpainting from the remaining image. The loss function we minimize favors the largest possible distance between this reconstructed background and the input image. This encourages the network to select regions that cannot be explained from their surrounding and are therefore salient. To handle the discrete nature of the proposals, we introduce a Monte Carlo-basee strategy to train our network. It operates discrete distribution, is unbiased, exhibits low variance, and is end-to-end trainable.

---

Under review as a conference paper at ICLR 2020
(a) YOLOv3
(b) Ours
(c)
(d) Ours
Figure 1: Domain specific detection and segmentation. Our self-supervised method detects the skier well, while YOLO trained on a general dataset does not generalize to this challenging domain. Similarly, MaskRCNN trained on a general dataset sometimes misses body parts and detects people in the background.
We tailor our training algorithm segmenting human motion from videos in which single subject moves. The constraints imposed by this well-defined, yet challenging and widely studied task, allow us generalize arbitrary, dynamic camera motion. As consequence, lets perform end-to-end training without the bootstrapping Croitoru et al. (2019) and intermediate optical flow supervision Papazoglou & Ferrari (2013); Jain et al. (2017p: Tokmakov et al. (2017).
We demonstrate the effectiveness of our unsupervised method on several datasets captured with increasingly mobile cameras, ranging from static to pan-tilt-zoom and hand-held. We will show that our approach applies to images acquired in conditions significantly more general than those of standard benchmarks, without requiring any manual annotation. Thus, as shown in Fig. [ approaches the quality and sometimes outperforms state-of-the-ar detectors that have been trained on large annotated datasets. Retraining or fine-tuning these methods on this data could be done but would require supervision that is hard obtain, which makes self supervised approach attractive. We will make our code and ground truth segmentations for the existing Ski-PTZ-Dataset and the new Handheld190k dataset publicly available upon acceptance of the paper.
RELATED WORK
Most salient object detection and segmentation algorithms are fully-supervised (Redmon et al. 2016 He et al., 2017]] Song et al. 2018 Cheng et al.] 2017) and require large annotated datasets with paired images and labels. Our goal a purely self-supervised method that succeeds without segmentation and object bounding box annotations. Note that this differs from the so-called unsupervised object detection methods (Hu al. 2018 Liet al. 2018 Jain et al. 2017). which still require domain speccffc annotations at training but not at test time. We focus our discussion on self- and weakly- -euell--upervised methods with regard to the type of training data used and refer to (Koh & Kim 2017) for the
discussion of methods using hand-crafted optimization.
methods. A classical weakly-supervised example is the Hough Matching algorithm (Cho et al., 2013). It uses an object classification dataset and identifies foreground as the image regions that have re-occurring Hough features within images of the same class. Similar principles have been followed using deep networks trained for object detection (Wei et al. 2017]
Jain al. (017). optical flow (Tokmakov al., 2017), and object saliency (Li et al. 2018). These methods make the implicit assumption that the background varies across examples and can therefore be excluded as noise. This assumption violated in the targeted case of training on domain-specific images, where foreground and background are similar across examples.
Motion-based methods Given video sequences, the temporal information can be exploited by assuming that the background changes slowly (Barnich & Van Droogenbroeck 2011) and lin- early (Stretcu & Leordeanu, 2015). However, even static scene induces non-homogeneous de- formations under camera translation, and it can be difficult to handle all types of camera motion (pan, tilt, zoom) within a single video, and distinguish articulated human motion from background motion (Russell et al. 2014). Some of the resulting errors can corrected by iteratively refining the crude background subtraction results of Streteu & Leordeanu (2015) with ensemble of student and teacher networks (Croitoru et al. (018.2019) This, however, induces a strong dependence on the teacher used for bootstrapping.
2

---

Under review as a conference paper at ICLR 2020
Spitheeaed Suepat
Ieput Imarge
P
Grop
Bcssed foregraand
aedrguoe Bedeouand model
Figure 2: Method overview. Encoder-decoder network (S) with attention mechanism defined by proposal-basee detection (D) and spatial transformers (T, 1). Via the use of an inpainting network (I), approach makes possible to train this network in entirely self- sepf-super manner unknown scenes with a moving background and captured with a hand-held camera.
Our approach is conceptually closely related to VidcoPCA (Stretcu & Leordeanu 2015). which models the background as the part of the scene that can be explained by a low-dimensional linear basis. This implicitly assumes that the foreground is harder model than the background and can therefore be separated as the non-linear residual. Here, instead of using motion cues, we propose to rely on the predictability of image patches from their spatial neighborhood.
Self-supervised Methods. Most similar to our approach are the superviid ones to object-
detection (Eslami al. 2016 Crawford & Pineau 2 Rhodin et al. 2019 Bielski & Favaro 2019)
that complement auto-encoder networks by an attention mechanism. These methods first detect one
several bounding boxes whose content extracted using a spatial transformer (Jaderberg et al.
2015). This content is then passed through an auto-encoder and re-composited with a background. In (Rhodin al. 2019), the background is assumed to be static and in (Eslami al. 2016 Crawford & Pineau 2019) even single colored, severe restriction in practice. Crawford & Pineau (2019)
use proposal- based network similar to ours, but resort approximating the proposal distribution
with a continuous one to make the model differentiable. Here, we demonstrate that much simpler importance sampling sufficient. (Bielski & Favaro (209)), generative model relies on the assumption that the image region strictly covering the salient object can be subject to random shifts without affecting the realism of the scene. However, this method can casily be deceived by other background objects whose random displacement can still yield realistic images. By contrast to all of these methods, our approach works with images acquired using a moving camera and given an arbitrarily colored background.
In addition to object detection, the algorithm of Rhodin et al. (219)) also returns instance segmentation masks by reasoning about the extent and depth ordering of multiple people in multi-camera scene. However, this requires multiple static cameras and a static background at training time, as does the approach Baqué et al. (2017) that performs instance segmentation in crowded scenes.
3
METHOD
Our goal is learn salient person detector and segmentor from unlabeled videos acquired in as generic a setup as possible, including using hand-held cameras. At inference time, our algorithm takes a single image I E WW' as input and outputs bounding box bm R-. expressed a center location, width, and height, and a segmentation mask S E 1228 128 within that window. This is achieved by the multi-stage process visualized in Fig. 2 first network predicts set of C' candidate object locations (b=) and corresponding probabilities (p=)_ 1- To this end, we use a fully-convolutiinall architecture that divides the image into a grid, with each cell yielding one probability, Pe. and an offset from the cell center to compute be. similarly to YOLO Redmon et al. (2016). Second, the bn with highest probability Pm 2 Pe Vc, is chosen and its content is decoded into foreground R 128x 128 segmentation mask S, and background B E RW x with separate encoder-decoder branches.
3

---

Under review as a conference paper at ICLR 2020
3.1 SUPEVIIEEVD TRAINING WITH KNOWN BACKGROUND
Given set of unlabeled images (I.) our goal is to train a neural network ((I) ((b.), (pe))== to propose suitable bounding box candidates and the corresponding probabilities to select the b.. that contains object. Because object locations
unknown, we do this with autoencoder objective, L (F(I, b,, B), I). that measures how well the autoencoder F reproduces the input image I on top of background B, with the attention on be- As in (Crawford & Pineau 2019 Rhodin al. 2019), this is implemented with a spatial transformer T that crops the area of interest, followed by a bottle-neck autoencoder S that produces the foreground and segmentation mask, and second spatial transformer T-1 that undoes the cropping and blends the synthesized foreground and background. We write
F(I. be, B) = (T) Bo - (T-) with ((T(I, + (I.S),
(1)
where o is the elementwise multiplication. The loss L can be least-square distance between all pixel values, alternatives are discussed at the end of the section. Because the attention window b selects part of the image for decoding, this loss encourages DD to focus on the object so as to model the foreground on top of B. Moreover, the autoencoder only approximates the actual image, forcing the segmentation mask to contain only the parts not captured by the background.
For now, assume B to be known, but will remove this assumption later. In this setting, derive a probabilistic formulation in which not a single, but multiple candidates can give rise to plausible reconstructions. We then reason about the expected loss across all likely candidates,
(I) E. (L (F(L, b.,, D), with p, where p(c) Pe.
where E. denotes the expectation
drawn from the categorical proposal distribution output by the network DII. We optimize Eq. (2) with stochastic gradient descent and mini-batches of (1 Note that minimizing this objective will jointly optimize the detection network D that generates and stochastically selects proposals, and the synthesis network S that models the object appearance and the segmentation mask.
Because we have a finite set of candidates, Eq. sum over all candidates as
2 can be expressed deterministically as a weighted
O(I) = (FL.
This objective is explicit function of p and could be optimized by gradient descent. However, this sum over C' forward passes is inefficient to evaluate in practice when C' is large. For example, in our experiments C' = 64, which does not fit in memory. This also observed by Crawford & Pineau (2019), who resorted to using a continuous approximation of the discrete distribution p to facilitate end-to-end training. Here, we propose a simpler alternative, exploiting Monte Carlo and importance sampling, which provides an unbiased estimator with low variance.
Monte Carlo (MC) sampling. In principle, the expectation in Eq. (2) could be estimated by sampling a small set of J candidate cells from p. Unfortunately, sampling from such discrete distribution is not differentiable with respect to its parameters, which precludes end-to-end gradient- based optimization. Instead of sampling according to p. we can rewrite Eq. (2) using an arbitrary distribution q, by reweighting with the quotient of both distributions. That is,
plc) L LF(II b., B). with ggg. ((())
O(I) E,
(4)
This change of distribution and relative weighting holds for any two probability distributions, explained in the appendix. In practice, we approximate the expectation on a mini-batch, with a single sample drawn from q per image.
as
Importance sampling. While moving the distribution into the expectation sum provides differen- tiability, it comes with the drawback of a potentially large variance, i.e., high approximation error for few samples. For instance, by choosing q be the uniform sampling distribution U1 most of the uniformly drawn samples will have a low probability in p and therefore negligible influence. To reduce this variance, we leverage importance sampling and set the sampling distribution g to be
4

---

Under review as a conference paper at ICLR 2020
similar or equivalent to p. To prevent division by very small values that could lead to numerical instability, we define the new distribution 9 as
g(c) = p(c) Cc)
(5)
As side effect, controls the probability that unlikely case is chosen, which induces a form of exploration that is helpful in the early training stages of the network. Note that the fraction p(c)
<(c)
cancels numerically for g(c) =(c).. However, while values cancel, their derivatives do not; the differentiability of pis is maintained as the sampling distribution g must be treated as constant. In this case, the gradient of Eq. equals that of the likelihood ratio method (Glynn] 1990) used in the REINFORCE algorithm (Williams 1992]. We favor the MC interpretation that explains the effect of the gradient quotient as importance sampling and generalizes to arbitrary q. While the relation is known for a long time (Glynn 1990), is overlooked in the recent literature.
3.2 SELF-SUPERVISED TRAINING WITH MOVING BACKGROUND
Having derived an efficient training scheme for proposal-basee segmentation when B is given, we would like to address the moving camera scenario, where it is not given, by reducing this more difficult case to the former. This can be achieved by predicting the background image, which, in the absence of prior shape and appearance information of the foreground, is even harder than segmentation.
To overcome this difficulty, we redefine the training strategy and replace background prediction with the simpler task of inpainting local regions, which can easily be trained by removing region and predicting it from its immediate surrounding (Pathak et al., 2016 Yu et af.] 2018). Under the assumption that the foreground object moves, network, I, trained on this self-supervised inpainting task would not manage to reconstruct the foreground objects fully removed in the input because the surrounding background gives no cues of their presence. We therefore cast foreground segmentation as the task of finding the area b. that, when inpainted, yields the largest image reconstruction error.
Instead of doing an expensive search at inference time, we train D to predict the foreground location. We search probabilistically by sampling likely cases according to the current distribution p and write
G(I) = -E. L(I I.). with c ~ p.
(6)
a(b.)
where I takes the image and the region b, to inpaint as input, is a pixel loss, a(b)) normalizes the loss by the window area, and p and b. are computed with D as before. We use the same importance sampling strategy to optimize the discrete distribution, but opposed to the foreground objective, we use the negative expectation. This negative reconstruction error encourages the selection of those regions were the true image is dissimilar to the reconstructed background when minimizing Eq. (6).
Training strategy. Unfortunately, the objective G has trivial solutions. It favors locations with high error density, irrespectively of their size, as illustrated in Fig. B[b). Alternatively, removing the normalization by the area favors erasing extensively large regions, containing an object or not, because larger number of reconstructed pixels lead to higher inpainting errors, as shown in Fig. 3
To overcome these degenerate cases, we combine the new background objective G with the foreground objective O of Eq. 4. substituting the known B with the learned inpainting III b.). The reason behind this is that these two terms are complementary: While G prefers locations that cover the object neither precisely nor entirely, O favors a tight fit over partial coverage but has trivial solution when b. is on a background region, i.e., not covering the object and having nothing to encode.
Finding a balance between these two adversarial objectives by relative weighting turned out to be difficult, if not impossible. Instead, we separate them such that their influence on the individual network components is mutually exclusive. Probabilities Pe are only optimized according to G. Therefore, O can not impose bias to background regions where has a trivial solution. In turn, b, is optimized only by O to find tight fit without the bias of G to excessively large or small proposals b.. Also is solely optimized by O to output the best possible reconstruction, instead of the largest distance to the background as induced by G. The separation can conveniently be computed in single forward-backwardh pass by treating the excluded variables as constants in the respective objectives, that means cutting of gradient flow through them.
Note that the stable training through effective separation into coarse and fine localization of adversarial termsis only possible with the chosen proposal -based detection framework; not with direct regression.
5

---

Under review as a conference paper at ICLR 2020
(a)
(c)
Figure 3: Ablation study on H36M. (a) Uniform sampling does not converge. (b) Joint training of O and G (c) only G (d) direct regression of single bounding box using O and G (e) Ours.
Implementation details. Since no labels for intermediate supervision are available, naive end-to- end training is difficult. To counteract this, we use ImageNet-trained weights for initialization; rely on perceptual loss on top of the per pixel L2 loss for L: exploit the Focal Spatial Transformers (FST) of Rhodin et al. (2019) to speed up convergence; and scale the erased region in I to be 1.1 the size of that predicted by D to increase the chances of covering the object. Moreover, we limit the location offsets to 1.5 the cell width and discard those outside the image. In addition, we rely on L2 priors on the output of DD and a v-shaped prior I., on S that stabilizes early training iterations by pushing the solution at the beginning of the training be larger than a value X, yet sparse when exceeding this threshold. We performed a grid search on the relative weights of the terms, the offset limits, and The pixel reconstruction and perceptual losses are weighted 1:2, and the priors have a weight of 0.1. 1, and 0.1, respectively, to compensate for their different magnitudes. Additional details are given in the appendix.
Regarding the inpainting network, using an off-the- inpainting method leads to hallucinated objects. Therefore, we train the inpainting network in self-supervised manner on the available input videos. Details are given in the appendix, Section A.1.
4 EXPERIMENTS
In this section, evaluate approach to self supervised salient object detection and segmentation. Note that our algorithm works on single images at inference time and requires the background inpainting model only training time. Our focus is people-detection. Nevertheless the method could generalize to arbitrary objects, as long as videos or picture collections of a single object in front of the same scene are available. We first demonstrate the of method in controlled environment with static background, to compare it to a state-of-the-- art self-supervised approach. Then, we introduce skiing footage acquired using PTZ-cameras and footage of people performing 14 everyday activities recorded using hand-held cameras to demonstrate that existing supervised methods that do well in the controlled environment struggle adapt such challenging conditions, whereas our approach delivers promising results. We provide additional qualitative results, experiments on the loss function, and details on the parameter search in the appendix material.
4.1 PEOPLE IN A CONTROLLED ENVIRONMENT
We compare our method against mate-of-the-ars ones on the Human3.6m dataset (Ionescu et al. 2014) that comprises 3.6 million frames and 15 motion classes. It features 5 subjects for training and 2 for validation, seen from different viewpoints against a static background and with good illumination.
Comparative Results. On the left side of Table T we compare our detection accuracy to that of very recent self-supervised deep learning method (Rhodin et al. (019), using the mean detection precision (mAP), the mean precision of having an intersection-over-unee Our slightly lower accuracy stems from not explicitly assuming static background, which Rhodin et al. (2199) do. While valid a lab, this assumption results in total failure in outdoor scenes with moving backgrounds, such as those discussed the next sections.
(IoU) of more than 50%.
Ablation Study. Here we show that our model choices for training and probabilistic inference are important. Using uniform sampling instead of importance sampling, does not converge, as shown in Fig. B(). Fig. 3(bb illustrates that joint training of D with O and G. instead of our separate one, produces bounding boxes that are too large. Fig. 3(c) shows that using only the background objective leads to small detections that miss the subject and (d) that direct regression without multiple

---

Under review as a conference paper at ICLR 2020
(a) Input
(b) Detection
(c) Segmentation
(a) Input
(b) Detection
(c) Segmentation
Figure 4: Multi-person detection and segmentation results, generated by sampling our model
multiple times. As the model is trained on single persons this only works for non non-intennn non-intersecting cases.
(a) Detection
(b) Segmentation
(a) Detection
(b) Segmentation
Figure 5: Qualitative results Ski-PTZ-camera and Handheld190k. Example results the test images. (a) The detection results show the predicted bounding box with red dashed lines, the relative confidence of the grid cells with blue dots and the bounding box center offset with green lines. (b) Soft segmentation mask predictions. Note that in the second row, the moving clouds are not segmented but the shadow the person can be included.
candidates diverges. These failure cases representative of the behavior the whole dataset. Big.Bio demonstrates that our full model using the separate training strategy and importance sampling accurately detect the person and estimate tighter bounding boxes.
Multiple people. Although our focus is on handling single objects or persons, our probabilistic framework can handle several at test time by sampling more than once. Fig. 4 shows the predicted cell probability as blue dots whose size is proportional to probability. The fully-convolutionall architecture operates locally and thereby predicts a high person probability close to both subjects. As a result, both the detection and segmentation results remain accurate as long as the individuals are sufficiently separated.
4.2 SKIERS FILMED USING A PTZ- CAMERSS
We now turn to the out-of-the the ordinary motions of six skiers on a slalom course featured in the Ski-PTZ-Dataset (Rhodin et al. 2018b). The six skiers split four/one/one to form training, validation, and test sets, with, respectively, 7800, 1818 and 1908 frames. The intrinsic and extrinsic parameters of the pan-tilt-zoom cameras constantly adjusted to follow the skier. As result, nothing is static in the images, the background changes quickly, and there are additional people standing as part of the background. We use the full image as input, evaluate detection accuracy in relation to the available 2D pose annotation, and segmentation accuracy by manually segmenting frames from each of the six cameras which add up to 108 frames for two test sequences. For the hyperparameter selection we use the manually segmented 3 frames from each of six cameras which total 36 frames for two validation sequences.
In Table I we evaluate detection accuracy. Note that our method delivers an mAPo.s score that is significantly better than that of the general YOLO Redmon et al. (016) detector trained on MSCOCO.
In T we compare our approach to several state-of-the-arr art segmentation baselines in terms of precision, recall, F-, and J-measure as defined (Pont-Tuset et al. 2017). To be fair, we compensate for different segmentation masks quantification levels by grid search (at 0.05 intervals) to select the best threshold in terms of J-measure for each method. Interestingly. MaskRCNN trained on a large generic dataset is outperformed by ARP this dataset. Without using any object localization data, our method is on par with MaskRCNN and close to weakly supervised methods that train on large datasets with motion boundary and segmentation mask annotations. More importantly, approach

---

Under review as a conference paper at ICLR 2020
Bielski & Favaro (019
(b) Ours
Figure Reconstructed scene and mask generated by Bielski & Favaro (2019) and our method on training examples. On the Ski-PTZ-Dataset the poles and snow patches are segmented as foreground. the Handheld190) dataset, the masks contain the foreground subject together with the ground they are standing on.
H366 dataset
Sli-PTT-Dataset
Method
mAP
Method
mAP
NSD Rhodin et al. 2019
0,710
YOLOv3 Redmon al. 2016
0.155
Ours
0.580
Ours
0.278
Table Defection results on the H36M and Ski-PTZ-Dataset datasets. They are expressed in terms of m.P,.s, the mean probability of having an pntersection-ove-umiiii (IOU) of more than 50%.
outperforms the self-supervised one of Stretcu Leordeanu 2013 While our and J-measures are slightly lower than that of (Croitoru et al. 2019). part of this difference can be attributed to Croitoru al. (2019) using a segmentation mask discriminator that is trained on the combination of the ImageNet VID and YouTube Objects datasets. Albeit also trained in self-supervised fashion, thereby leverages additional information.
In Fig. 6 we show the scenes reconstructed by Bielski & Favaro (019)) along with the resulting segmentation masks. Note that their generative model fails to segment the foreground object alone and instead segments background objects and sometimes even the ground. This method relies the property that foreground regions can undergo random perturbations without altering the realism of the scene. However, in the Ski-PTZZDDtaset, some background objects, such poles, also satisfy this property, and the generator can choose to keep these regions. Since the results on the training samples are very poor, we do not provide any quantitative results of this work on the test data.
Further qualitative results are shown in Fig. The probability distribution, visualized as blue dots that increase in magnitude with the predicted likelihood, show clear peaks the persons. The limitations include slightly blurred and bleeding masks and occasional false positives. reducing precision.
We investigate the effectiveness of different mask priors and ImageNet pre-training on the validation part of the SKi-PTZ-Dataase in the appendix. Our f, prior achieves the highest scores in all measures with consistently reliable results. They demonstrate that imposing regularization on the segmentation masks allows us to obtain sharper masks, removing the noise around the foreground object.
4.3 DAILY ACTIVITIES CAPTURED USING HAND-HELD CAMERAS
We introduce test sequences, comprising 120 855, 23 076 and 46 326 images, respectively, with a single actor performing actions mimicking those in H36M. We manually annotated 112 frames in the validation and 240 frames the test sequence to provide ground-truth segmentation masks which we believe will be useful for evaluating other self- and weakly-supervin methods. The camera operators moved
Handheld1904 dataset that features three training, one validation and one
Ssi-PTZ-Dataee
Handheld190k
Method
Precision
Recall
F Messure
J Measure
Precision
Recall
measure
I measure
MasKRCNN Ho et
2017
0.75
0.65
0.68
0.65
0.93
0.88
0.90
0.77
ARM Kim
0.94
0.83
0.73
0.67
VideoPCA Strelcu Leoedaau 2015
0.49
0.84
0.61
0.56
0.35
0.87
0.49
0.47
Unsup-Dilan Dillto-NNa Cronrod 2019
0.74
0.76
0.74
0.67
Ours"
0.75
0.56
0.63
0.56
0.80
0.74
0.76
0.68
Table Segmentation results on the Ski-PTZ-Dataser and Handheld190k dataset. Ours exceeds or is on par with the self-supervised methods (marked with *), and approaches the accuracy of weakly-supervised (marked with ") and fully-supervised methods (marked with d).
8

---

Under review as a conference paper at ICLR 2020
laterally, to test robustness to camera translation and hand-held rotation. We provide examples of our detection and segmentation results in Fig. [ more are given in the appendix. Our method is robust to the undirected camera motion and to dynamic background motion, such branches swinging in the wind and clouds moving, and to salient textures in the background, such as that of the house facade.
To perform quantitative comparison, we use 240 manually segmented test images taken from different motion classes with the subject in many different poses. We evaluate against the same methods using the same quantities as for skiing in Table In this scenario, MaskRCNN yields the highest scores, which is not surprising as the tested sequences are similar to its training set MSCOCO. is followed by our work in F- and J-measure. Despite using discriminator that is trained in an unsupervised fashion on another, larger dataset, (Croitoru et al. (2019) performs comparably to ours. Contrary to Ski-PTZZ-Datasee and have larger margin on VideoPCA [Stretcu & Leordeanu (201)). Both of these methods often fail in separating the non-homogeneously moving background due to the hand-held camera motion.
results, we outperform ARP Koh & Kim (2017) in F- and J-measure
CONCLUSION
We have proposed a self-supervised method for object detection and segmentation that lends itself for application in domains where general purpose detectors fail. Our core contributions are the Monte Carlo-based optimization of proposal-based detection, new foreground and background objectives, and their joint training on unlabeled videos captured by static, rotating and handheld cameras. Our experiments demonstrate that, even if trained only on single persons, our approach generalizes to multi-person detection, as long as the persons solutions [Barnich & Van Droogenbroeck] (2017 1); Russell et al. (2014); Croitoru et al. (2018), our approach does not exploit temporal cues. In the future, we will integrate temporal dependencies explicitly, which will facilitate addressing the scenario where multiple people interact closely, by incorporating thysics-inspired constraints enforcing plausible motion.
sufficiently separated. In contrast many existing
REFERENCES
Baqué, F. Fleuret, and P. Fua. Deep Occlusion Reasoning for Multi-Camera Multi- Multi-Target Detection. In International Conference on Computer Vision, 2017.
Olivier Barnich and Marc Van Droogenbroeck. Vibe: universal background subtraction algorithm for video sequences. IEEE Transactions on Image processing, 1006) 2011.
Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative modelsn. NeurlPS, 2019.
Jingchun Cheng, Yi-Hsuan Tsai, Shengjin Wang, and Ming- Hsuan Yang. Segflow: Joint learning for video object segmentation and optical flow. In Proceedings of the IEEE international conference on computer vision, pp. 686-695, 2017.
Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. Unsupervised object discovery and localization in the wild: Part-based based matching with bottom-up region proposals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1201-1210, 2015.
E. Crawford and J. Pincau. Spatially invariant unsupervised object detection with convolutional neural networks.
In Conference Artificial Intelligence, 2019.
Ioana Croitoru, Simion-Vias Bogolin, and Marius Leordeanu. Unsupervised learning of foreground object detection. arXiv preprini (tt.:1800004599 2018
Ioana Croitoru, Simion- Vlad Bogolin, and Marius Leordeanu. Unsupervised learning of foreground object segmentation. International Journal of Computer Vision, pp. 1-24, 2019.
S.M.A. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K. Kavukcuoglu, and G. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
P.M. Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of ACM, 33(10): 75-84, 1990.
K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask R-CNN. In International Conference on Computer Vision, 2017.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Yuan- Ting Hu, Jia-Bin Huang, and Alexander G Schwing. Unsupervised video object segmentation using motion saliency-guidee apatio-temports propagation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 786-808 8020802, 2018.
Eddy 1lg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2017.
C. lonescu, I. Papava, Olaru, and C. Sminchisescu. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial Transformer Networks. In Advances in Neural Information Processing Systems, pp. 2017-2025, 2015.
Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion appearance for fully automatic segmentation of generic objects in videos. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp.
IEEE, 2017.
Yeong Jun Koh and Chang-Su Kim. Primary object segmentation in videos based region augmentation and reduction. In 2017 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 7417-7425. IEEE, 2017.
Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, and C-C Jay Kuo. Instance embedding transfer to unsupervised video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6526 6335,65352200 2018.
T-Y. Lin, M. Maire, S. Belongic, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C.I., Zitnick. Microsoft COCO: Common Ohjects in Context. In European Conference on Computer Vision, pp. 740-755, 2014.
Papazoglou and Ferrari. Fast Object Segmentation Unconstrained Video. In International Conference on Computer Vision, pp. 1777-1786 1784, 2013.
D. Pathak, P. Krähenbühl, J. Donahue, Darrell, and A. Efros. Context Encoders: Feature Learning by Inpainting. CoRR, hasp//604007779,,. 2016.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Sorkine-Homungg and Lue Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint arXiv:1704.00675 2017.
Redmon, S. Divvala, Girshick, and A. Farhadi. Only Look Once: Unified, Real-Time Object Detection. Conference on Computer Vision and Pattern Recognition, 2016.
H. Rhodin, M. Salzmann, and Fua. Unsupervised Geometry- Aware Representation for 3D Human Pose Estimation. In European Conference Computer Vision, 2018a.
H. Rhodin, J. Spoerri, I. Katircioglu, V. Constantin, F. Meyer, E. Moeller, M. Salzmann, and P. Fua. Learning Monocular 3D Human Pose Estimation from Multi- View Images. Conference on Computer Vision and Pattern Recognition, 2018b.
H. Rhodin, V. Constantin, I. Katircioglu, M. Salzmann, and Fua. Neural scene decomposition for multi-person motion capture. 2019.
O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Conference Medical Image Computing and Computer Assisted Intervention, pp. 234 241, 2015.
C. Russell, R. Yu, and Agapito. Video pop-up: Monocular 3d reconstruction of dynamic scenes. In European Conference on Computer Vision, pp. 583-598. Springer, 2014.
Hongmei Song, Wenguan Wang. Sanyuan Zhao, Jianbing Shen, and Kin-Man Lam. Pyramid dilated deeper convlstm for video salient object detection, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 715-731, 2018.
Otilia Stretcu and Marius Leordeanu. Multiple frames matching for object discovery in video. In BMVC, volume 1, pp. 3, 2015.
10

---

Under review as a conference paper at ICLR 2020
Pavel Tokmakov, Karteek Alahaari. and Cordelia Schmid. Emergence of object segmentation in perturbed generative models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Xiu-Shen Wei, Chen-Lin Zhang, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou. Unsupervised object discovery and co-localization by deep descriptor transforming. arXiv preprint arXiv: axXiv:/77.063997 2017.
R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, In Reinforcement Learning. 1992.
Zhirong Wu, Yuanjun Xiong. X Yu Stella, and Dahua Lin. Unsupervised feature learning via non paraameettiic instance discrimination. 2018.
Jiahui Yu. Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conferencee on Computer Vision and Pattern Recognition. pp. 2505-5514, 2018.
Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
A APPENDIX
In this document, we supply additional implementation details, qualitative results, and mathematical justification for the importance sampling transformations.
A.1 IMPLEMENTATION DETAILS
For efficiency, W and and and share parameters, except for their output layers. In the following, we explain the architecture of each sub-network.
Detection network. We predict one candidate relative each cell in a regular grid using fully- convolutional architecture similar YOLO Redmon al.] (2016). We use ResNet-18 backbone [H et al.] (2016), that reduces the input dimensionality by a factor 8, forming a low resolution grid of features, e.g., to spatial resolution 8 x 8 from 128 x 128. The feature size is set to 5; two for bounding box location. two for scale, and one for the probability. Each feature output represents one cell, the offset is predicted relative to the cell center sketched in Fig. (main document). The output of p is forced be positive and form proper distribution, with Σ predicting bounding boxes at the borders of the image, zero the outer cell probabilities.
1. by soft-max activation unit. To prevent this network from constantly
The synthesis network. has the form bottle-neck auto-encoder, based on publicly available implementation of Rhodin et al.) (2018a). The encoding part is a 50-layer residual network and weights are initialized with ones trained on ImageNet classification. The hidden layer is 728 dimensional, split into an 600 dimensional space that goes through two fully connected layers and 128 dimensional space that is replicated spatially 128 x X feature map to encode spatially invariant features. The decoding done with the second half of U-Net architecture Ronneberger et al. (2015) with 32, 64, 128, 256 feature channels in each stage. The final network layer outputs four feature maps, three predict the color image ī and one for the segmentation mask.
Discussion We use YOLO detection architecture combined with spatial transformers and syntesis network S. To improve inference speeds, one could merge D and S to share the same base network through Rol pooling in MaskRCNN. Nevertheless, the MC training approach is necessary, as the inpainting network can not share the same features for multiple bounding box locations since blacked-our imput region varies each b..
The inpainting network. I is 6 layer U-Net model with 8, 16, 32, 64, 128, 256 feature channels in each stage. The input to this network is image where the selected bounding box region is hidden and the output initially hidden patch being reconstructed. It trained independently from the rest of
the entire image the pipeline by feeding images with randomly occluded regions of varying sizes. In our full pipeline, the weights of the inpainting network are frozen and to remove the image evidence corresponding to the foreground person, the hidden patch in the input image the inpainting network is selected to be bounding box region expanded by 20% in both dimensions.
11

---

Under review as a conference paper at ICLR 2020
(b)
(d)
Figure 7: Off-the-shelf inpainting results, on skiing.
(a) Input image with the hidden middle part,
followed by inpainting with (b) Pathak et (2016), (c) Yu al. (2018) trained on ImageNet. (d) and
Yu et al. 2018) trained on Places2.
Figure 8: The capture setup of our in-house handheld posing dataset. The subject is recorded by three persons with handheld GoPro action cameras.
In principle, any off- she-thh--hhfffhce inpainting network trained on large and generic background datasets could be used. For instance, the methods of Pathak al. (2016); Yu et al. (2018) can produce very plausible results. However, domain -speiie images, they tend hallucinate objects as shown in Fig. and are therefore ill-suited our goal. Instead, we train I from scratch, by reconstructing randomly removed rectangular image regions, as shown in Fig. 2 Note that it is fine for this network not to generalize well to new scenes as it is not needed at test time.
Overall training All training stages performed single NVIDIA TITAN X (Pascal) GPU with Adam and a learning rate of Ic-3. First, the inpainting network optimized for 200k iterations and subsequently the complete network for additional 100k iterations. The decoding pars of the synthesis network uses a reduced learning rate of le-4, to prevent occasional diverging behavior. We use a batch size of 16 and an input image resolution of 640pxx x660px for the Ski-PTZ-camee dataset, 640px 360px for the in-house handheld posing dataset, and 500pxx500pp for H36M.
Importance sampling For the importance sampling function g we use = 0.001, which makes the method numerically stable while the probability of choosing random bounding box stays low, i.c., 6.4% for 64 cells.
A.2 Ski-PTZ-Dataset
In Table compare different mask priors and analyze the effect of using pre-trained weights. Although yields better segmentation masks than éz. might suppress mask values too strictly which causes convergence problems. This mitigated by our &u prior, which achieves the highest scores all measures with consistently reliable results. Our unsupervised training method suffices for obtaining correct results from scratch. Nevertheless, given our relatively small dataset, using pre-trained network significantly improves the stability of the training well the results. To this end use ImageNet pre-trained model since is both practical
12

---

Under review as a conference paper at ICLR 2020
Ski-PTZ-Dulust
Setting
Precision
l'Messure
price
prir
Imageeac pro-tainning
Unsgersisel pee-tzinnnn
2013
pilor
0.71
Table 3: Analysis of the mask prior effect and Imagenet pre- pre-training on the Ski-PTZ-Datase dataset. We demonstrate the influence of using mask priors to supress the noise surrounding the foreground object and have clear-cut masks. At the bottom part of the table we show the results of using random weights and features from Wu et al. (2018) instead using weights from Imagenet pre-training.
Ski-PTZ-Dataset
Setting
Recall
Precision
Measure
Measure
b 1..1.0 000000
0.62
0.35
0.44
0.43
b.0.1.055.. L
0.38
0.65
0.47
0.41
b 00,1=00055
0.37
0.65
0,47
0.43
b0..1.0.5) 60,x=0.11
0.38
0.62
0.47
0.43
b0.155,03]. tv,A=0.001
0.35
0.59
0.43
0.43
b00.15,0.5,, 6,,A=0.0022
0.50
0.69
0.57
0.51
b00.15,0.5], 6v,A=0.005
0.35
0.58
0.43
0.42
10.115.055,, 6v,
0.39
0.65
0.48
0.46
b 00220,0.3], f,,A=0.001
0.48
0.71
0.57
0.52
b 200.20.0.5,, be,A=0.000
0.44
0.64
0.52
0.45
(00.00.0.5. 4,,2=0.005
0.37
0.63
0.46
0.44
b 200.00.05) 4,,1=0001
0.37
0.59
0.45
0.42
b (0.25.0.5). 00,1=00001
0.39
0.58
0.46
0.43
b 100.25.0.5, f,,A= 0.020000255
0.40
0.62
0.48
0.44
b 6r,2
0.43
0.63
0.51
0.47
2 f0,100001
0.41
0.61
0.49
0.45
b |0.30,00.5 60,1=0.001
0.38
0.55
0.45
0.41
b b00.30.0.3]. 6v,A=0.0025
0.41
0.54
046
0.40
[0.30.0.5]. le,x=0.000
0.39
0.54
0.45
0.42
b 00330.0.5), f0,x=0.01
0.40
0.59
0.48
0.44
b (0..5.0.55, 00.0000
0.50
0.64
0.56
0.51
b (0035.0033 00020000025
0.44
0.62
051
0.46
b (0.35.0.5).
0.35
0.48
0.40
0.38
b 0.35,0.5)
x0,000.11
0.40
0.51
0.45
0.40
b (00
Ly t=,A=0.001
0.39
0.62
0.48
0.45
b (00.0,0.600
tu,2=0.001
0.48
0.68
0.56
0.49
b 060.000066))
f,A==0.003
0.38
0.61
0.46
0.46
b 200.20.0.700
t0,1=0.001
0.54
0.73
0.62
0.56
200.20,0.75] A0.000001
0.54
0.74
0.62
0.55
200200,0.0 00.000.001
0.52
0.77
0.62
0.56
Table 4: Hyper-parameter study on the Ski-PTZ-Datasel dataset. In this table we analyze the effectiveness of our hyper-parameter choice for the minimum and maximum bounding box sizes (given in square brackets) as well as the threshold À for the &, loss.
and the weights readily available for diverse architectures, although from different domain and task. To quantify the reliability of our method with regularizer, we repeated the Ski-PTZ-Datase experiment three times with the best-perfoming configuration and computed the mean and std: the Precision, Recall, F- and J-Measure are consistent. respectively, 0.54 = 0.02, 0.60 + 0.01, 0.74 + 0.02, 0.51 11 +: 0.01.
In Table 4 we compare the performance of method for different values of hyper-parameters, where the subscript corresponds the minimum and maximum size of the hounding box and À is defined Eq. 7
13

---

Under review as a conference paper at ICLR 2020
Txe
WH T (S)
Σ
Σ
- +2,
A.3 Handheld190k DATASET
The capture setup of our Handheld190k dataset is depicted in Fig. [ The dataset composed of five actors performing the same actions as those available in the H36M dataset lonescu et al. (2014), namely directions, discussion, eating, greeting, phone talk, posing, buying, sitting, sitting down, smoking. laking photo, waiting, walking, walking dog and walking in pair. We excluded lying on the floor actions, not making our actors lic the dirt. The data from three actors compose our training set and the other two form the test set. The data was obtained using 3 GoProó cameras recording FullHD videos at 30 FPS in lincar lens mode, As seen in the Fig. B it is an outside recording. For the entire duration, the cameras were subject to lateral movement and varying hand-held rotation. The motion stablization of the GoPros deactivated the time of the recording.
A.4 QUALITATIVE RESULTS
We provide additional examples of detection and segmentation results the Handheld190k test in Fig. 9 Fig. also shows results from the ski set. Additional segmentations the established H36M dataset are given Fig. I0 Notably, our method robust to the undirected camera motion and to dynamic background motion, and works equally well for the very different domains of skiing and every-day activitics.
A.5 IMPORTANCE SAMPLING THEORY
In the following, we give an explanation for the change of distribution when computing an expectation. Let p. be two discrete probability distributions and f(c) an arbitrary function of c. Reasoning about the limit towards infinitely many samples ej drawn from q(). we derive
Eq(e)
p(c) s(e) f(e)
lim J
J
1
plc,) q(c))
f(G))
5=1
Ejsr 16,
p(c)
lim
(())
1-00
J
a(c)
cml
(pc)
Σ jg(c)
( s(e)
((c))
Σ p(o)((E)
= Eptc) [f(c)]
(8)
where J is the number of samples drawn and Le, me one if c, equals c. In the second line, we exploit that we have a finite number of classes C. Each sample must fall into one of them and the probability of coming from cell is (()).
This relation provides with tool change the sample distribution for expectations. Next, we analyze variance of such estimator depending on the chosen sampling distribution.
A.6 IMPORTANCE SAMPLING VARIANCE
The variance of an estimator gives us a measure for the expected accuracy after limited number of samples. We use it to compare the three candidates discussed in the main document. The variance of estimating the objective O with a monte carlo sum over Cl, CJ samples drawn independently from is
Var [O(1) z Var } EE(FLL)))))))
1
Var [L [L (F(I.)).
J 3=1
j Var (F(I,c,))
14

---

Under review as a conference paper at ICLR 2020
1
f
(a) Detection
(b) Inpainting
(c) Synthesized image
(d) Segmentation
(e) Groundtruth
Figure Detection and segmentation results on the test subjects of Ski-PTZ-camera and Hand- held190k dataset. (a) Detection result. The blue dots coincide with the grid cell centers and their size indicates the confidence of the bounding box proposals. The selected bounding box illustrated with red dashed line and the center of the grid cell yielding this proposal is connected to the center of the red box through the green line. (b) The inpainting result where the region inside the detected bounding box is reconstructed by our inpainting network (only needed for training). with the predicted foreground and background regions combined. (d) Segmentation mask result. (e) Groundtruth segmentation mask.
(c) The synthesized image
15

---

Under review as a conference paper at ICLR 2020
Rhodin al. Rhodin et al. (2019)
Ours
(a) Input
(b) Detection
(c) Cropped
(d) Segmentation
Figure 10: Detection and segmentation results H36M. Results match in quality with those from
Rhodin et al. (2019). with a slight bleeding due to not having perfect background prediction oracle.
Here Var (FLL, o3), I)] is the variance over the random variable c, and we utilized the identity Var[az] a² Var[ar] and the independence of samples. The variance reduces linearly with the number of samples and is proportional that of I.-).
contrast, using uniform sampling (Eq. main document, with g U.). yields quadratic variance growth with the number of cells,
ples) LF (M((e))
Var (T(T)
Var
J
1
Var
J
Var (FL, 1)).
(10)
From which we conclude that uniform sampling leads to a higher variance, since the remaining Var(p(-) term is not expected to improve on Var(I.())
With importance sampling according to (Eq. main document) the variance is,
Var [O()] 3 Var
p(c) glc)
(F(L,)
(11)
If q < it is equivalent to the one of Eq. 9 In general. should be constructed to minimize Eq. 11 In our case, F depends each individual image and cell it thereby difficult impose assumptions reduce the variance of L. without evaluating for each c. Therefore, setting g p good choice from the perspective of variance reduction.
A.7 ADDITIONAL COMPARISONS RELATED WORK
Motion-basse based approaches are prone to fail when there is no or too complex motion information to separate foreground and background and in textureless areas. These failure cases occur in all our scenarios. as shown in the optical flow images in Fig. generated from our datasets. To this extend, we excluded motion cues in this study. We planning integrating their complementary merits the future.
16

---

Under review as a conference paper at ICLR 2020
(a) H36M (b) H36M
(c) Ski-PTZ
(d) Ski-PTZ
(e) Handheld190k
(f) Handheld190k
Figure 11: Optical flow failure cases. Top: Input images with too complex motion information.
Bottom: Corresponding optical flow images generated using llg et al. al.] (2017). The white portions in the flow show the regions that stationary relative to the camera. Multiple colors in the flow indicate different motion directions which can not be easily segmented into foreground and background.
Figure 12: Examples heatmaps and detection results from Zhou et al. (0016) on H36M. The
heatmap generated by the ImageNet pre-trained model has multiple high activations corresponding to a given ImageNet category. Since the candidate bounding boxes are selected such as to overlay the connected components of the heatmap, they cover large area. Therefore using these candidates to efficiently localize the salient object can often fail.
We also investigate the possibility of using pre-trained localization network from Zhou et al 2016 and obtain the heatmaps and the corresponding bounding box predictions for H36M S9 and SI1 test images using their Googl .eNet-CMM pre-trained model ImageNet. As the output heatmaps in Fig. T2 show, the pre-training on ImageNet alone is not reliable have accurate detections. This is mainly due the multiple activated regions that yield very big bounding boxes covering not just the human subject but also the other objects in the scene matching with the ImageNet classes. Among all the bounding box predictions that Zhou et al.) (2016) generates, we select the one yielding the highest IOU score against the groundtruth bounding box corresponding 0.1 which is significantly lower than our mAPu.s score of 0.58. This gap indicates that doing the detection using only their pre-trained model is reliable and selection of the best bounding box among the candidates unfeasible in self supervueeeeieed setting. Our model can accurately detect the human subject without needing an ImageNet pre-tree re-training in the detection component. Regarding the pre-training on the encoder part of S we refer the reader to Table Comparing the ImageNet pre-trained model (in the fourth row) with the unsupervised pre-trained model (in the last row), the unsupervised setup shows only a small performance drop; pre-training helps hut is not essential.
that frame. We report that Zhou al. 2016) has mAPu.
Another part of our approach where we can provide additional comparison the sampling method. To explore an alternative strategy to Monte-Care based sampling, we tried replacing the importance sampling part in method with the categorical reparameterization used in Crawford & Pincau 2019). Since both strategies approximate the objective, they should lead very similar outcomes with possible difference in the
17

---

Under review as a conference paper at ICLR 2020
Figure 13: Qualitative results comparing the bounding box detections obtained through our im- portance sampling strategy against the Gumbel-Softmax categorical reparameterization with temperature hyperparameter
0.1. Top: Ours. Bottom: Categorical reparameterization. The results are taken from the same iteration for both models.
convergence speed. To this end we used Gumbel-Softmn distribution and tried this estimator with several different temperature values. Our experiments show that Gumbel-Sofee -Sftmax based categorical reparameterization didn't 't lead to faster convergence shown Fig. 13 It can be seen that for the same iteration from which the results taken the importance sampling based model has already converged in terms of detection whereas the categorical reparameterization hasn't 't converged yet. This might partly due to the high value Gumbel noise added to the log probabilities. However, to have more solid claim, principled grid search for hyperparameters such as the temperature is necessary. We will provide the exact comparison in our final version after conducting extensive hyperparameter search. Overall importance sampling approach is simpler compared to their approach and has the advantage being an unbiased estimator. In addition that it does not need custom layers that behave diffcrently in the forward and backwards passes during optimization, which the case the Gumbel-Softmax categorical reparamcterization.
18


Reviews:
[{"review_id": "HyloanZEcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a new self-supervised proposal-based approach for object detection and segmentation. The author introduces a Monte Carlo-based optimization to solve the inefficiency problem in the discrete proposal-based forward process defined in (Crawford and Pineau 2019). Also, the paper redefines the decoder part for self-supervised from minimizing reconstruction loss with background segmentation to maximize reconstruction error with learning a foreground segmentation.  The method is then verified with a suite of experiments for people-detection on video datasets.\n\nThe main benefit over many previous unsupervised object detection/segmentation approaches is that they did not make use of optical flow or other readily available cues during training. However, given that the framework directly came from (Crawford & Pineau 2019), and the only change is from variational inference to an importance-sampling (MC) approach. This would be fine if it is verified in experiments, however, the experiments did not show any comparison w.r.t. (Crawford & Pineau 2019) hence we have no way of understanding what is the relative performance w.r.t. that baseline approach.\n\nBesides, in all the experiments a single object is in the view. How does the method perform in images where multiple objects are in the view?\n\nA little bit of a philosophical question is whether this a problem worth pursuing as well. For self-supervised motion estimation (e.g. optical flow), it is clear why we want to do that. However, the current type of algorithm is dependent on the assumption that the background follows relatively consistent textures, this may not necessarily be true in practice, and hence the application could be quite limited. Many previous unsupervised video object segmentation methods make use of optical flow and boundary detection, which I thought are OK cues to be used, especially when both can be learned in a self-supervised manner. This is not entirely related to the assessment, but I would still like to hear what the authors think.\n\nMinor:\nIn the paragraph after 'Training strategy'(Section 3.2, Page 5), is it 'the foreground objective O of Eq. (2)' or 'Eq.(4)'?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HklAE1a2Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a method to self-supervised train a model for object detection/segmentation. The idea is that the background is easy to reconstruct while the foreground/object is hard. Experiments demonstrate the effectiveness of the proposed methods.\n\nHere are some high-level concerns.\n\n1. As mentioned in the \"Implementation details\", 'naive end-to-end training is difficult... we use ImageNet-trained weights for initialization'. This is worrisome to justify the effectiveness. It may be possible the imagenet-trained model has already captured salient objects. To justify the claims and effectiveness of the method, it should include a comparison with [R1], which demonstrates the possibility of doing detection with a pretrained model. Other work along this line should be also good reference.\n\n2. As a moving camera is available, it is also possible to segment background with frames through a 6DoF prediction on the camera, rotation and translation, e.g., [R2]. The supervision signal is from frame reconstruction through learning to predict both camera pose and pixel-level depth. This is also self-supervised learning. At least such a self-supervised trained model can act as an initialization.\n\nConsidering the above points, the paper does not appear compelling, due to lack of either careful claims or justification.\n\n\n[R1] Learning deep features for discriminative localization\n[R2] Unsupervised learning of depth and ego-motion from video", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BylC3XOcYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a self-supervised segmentation method, that learns from single-object videos by finding the region where it can segment an object, remove the entire bounding box around it, inpaint it, then finally put the object back. The loss is a balance between a reconstruction error and a negative inpainting error (high error means object is probably present, due to the weak correlation with the background).\n\nMy decision is Weak Accept. I like the method very much and think it’s a clever and well-executed algorithm. The reason for being Weak is because the experimental evidence could be stronger, especially comparing with Croitoru et al. and Rhodin et al. The paper leaves some open problems, inviting future work to be built on top of it (i.e. leveraging time more and handling multiple objects better). I think it should be accepted to promote such future work.\n\nMethod:\n\nThe method is interesting and clever. Similar efforts have been made, such as Bielski & Favaro and Crawford & Pineau. However, key contributions relax some of the contrived requirements of these past methods (e.g. simple foreground translation over background; requirement of plain background). Thanks to the inpainter, the importance sampler, and avoidance of collapsing into trivial solutions, this paper is able to put together a method that works on moving cameras and fairly complex scene semantics (still limited to one object though). Actually getting this to converge with only two loss terms balanced against each other is impressive. Having certain heads of the network trained only on one of the two terms seems to be a key contribution. \n\nExperimental results:\n\nThe comparison with Rhodin et al. on H36M is characterized as “slightly” lower, but I would call 71% vs. 58% a significant difference. Of course, I understand that Rhodin et al. relies on the static background, so this is not a fair comparison.\n\nSki-PTZ-Dataset should then offer a better comparison, but here the method struggles to compete with Croitoru et al. 2019, which has a 11-point higher F-measure. On Handheld190k, the dataset proposed in this paper, is where the method finally shines, but still only offers a 1-point F-measure improvement over Croitoru et al. That being said, considering how different the methods are, and how Croitoru et al. requires two-stage training, there are many benefits to this method. Croitoru et al. also relies on video to extract the object features, and this requirement is not as explicit in this work. Actually, that brings me to one question I had. The paper states that “as long as videos or picture collections of a single object in front of the same scene are available.” I didn’t quite understand why this must be trained on multiple images of the same scene. If the inpainter is general to any background scenery, couldn’t it work on single-images as well? The conclusion even says you do not use temporal cues.\n\nOther:\n\nI don’t think it’s that meaningful to include precision/recall in the tables. It is also not that meaningful to point out that your method’s precision is higher than that of Croitoru et al., when the F-measure is shy 11 points. The reason is because many points on the precision/recall could be constructed simply by applying for instance a gamma curve on the segmentation predictions. The high precision is clearly at the cost of a low recall, and another point on this tradeoff curve could be presented. This is why F-measure and average precision are much better.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a self-supervised segmentation method: building upon Crawford and Pineau 2019, this work adds a Monte-Carlo based training strategy to explore object proposals.
Reviewers found the method interesting and clever, but shared concerns about the lack of a better comparison to Crawford and Pineau, as well as generally a lack of care in comparisons to others, which were not satisfactorily addressed by authors response.
For these reasons, we recommend rejection.
Decision: Reject