MOVE: Unsupervised Movable Object Segmentation and Detection
Authors: Adam Bielski
Paolo Favaro
Venue: NeurIPS 2022 Accept (2022)
Keywords: Object Discovery
Saliency Detection
Object Segmentation
Object Detection
Self-Supervised Learning
Unsupervised Learning

Abstract: We introduce MOVE, a novel method to segment objects without any form of supervision. MOVE exploits the fact that foreground objects can be shifted locally relative to their initial position and result in realistic (undistorted) new images. This property allows us to train a segmentation model on a dataset of images without annotation and to achieve state of the art (SotA) performance on several evaluation datasets for unsupervised salient object detection and segmentation. In unsupervised single object discovery, MOVE gives an average CorLoc improvement of 7.2% over the SotA, and in unsupervised class-agnostic object detection it gives a relative AP improvement of 53% on average. Our approach is built on top of self-supervised features (e.g. from DINO or MAE), an inpainting network (based on the Masked AutoEncoder) and adversarial training.

MOVE: Unsupervised Movable Object Segmentation and Detection
Adam Bielski
Paolo Favaro University of Bern paolo favaro@unibe.co
University of Bern adam. bielski Cunibe.
Abstract
We introduce MOVE, a novel method to segment objects without any form of supervision. MOVE exploits the fact that foreground objects can shifted locally relative to their initial position and result in realistic (undistorted) new images. This property allows us to train a segmentation model on dataset images without annotation and to achieve state of the art (SotA) performance several evaluation datasets for unsupervised salient object detection and segmentation. unsuper- vised single object discovery, MOVE gives average CorLoc improvement of 7.2% over the SotA, and in unsupervised class-agnostic object detection it gives relative AP improvement of 53% on average. Our approach is built top of self-supervised features (e.g. from DINO or MAE), an inpainting network (based the Masked AutoEncoder) and adversarial training.
Introduction
Image segmentation and object detection are today mature and essential components in vision-based systems with applications medicine (3]. just to name few. A major challenge in building and deploying such components at scale is that they require costly and cone-con human annotation. This has motivated efforts in self-supervised learning (SSL) 4. The aim of SSL is to learn general-purpose image representations from large unlabeled datasets that can be fine-tuned to different downstream tasks with small annotated datasets. While SSL. methods have also been fine-tuned for image segmentation since their inception, it only with the recent state of the art (SotA) methods, such as DINO 4 and Dense Contrastive Learning (6), that clear and strong link to object segmentation has been observed. This has led to several methods for salient object detection built on top of SSL features (7H12).
wide range of fields including automotive [1. agriculture [2], and
Most prior work based on SSL features defines some form of clustering by either using attention maps 97-9 or similarity graphs 12). In this work, we take a quite different direction. Rather than directly clustering the features. train network map them to segmentation mask. As supervision signal we use the movability of objects, i.e. whether they can be locally shifted realistic manner. We call method MOVE This property holds for objects in the foreground, as they occlude all other objects in the scene. This basic idea has already been exploited in prior work with relative success [9]. Nonetheless, here introduce novel formulation based on movability that yields significant performance boost across several datasets for salient object detection.
In our approach, it is not necessary to move objects far from their initial location or to other images [4 [6] and thus we do not have to handle the context mismatch. It is also not necessary to employ models to generate entire scenes [15] 20), which can be challenging to train. Our working principle exploits observations also made by [17 They point out that the correct mask maximizes the inpainting error both for the background and the foreground. However, rather than using the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

prodicted music
inceinted background
ahiteed foreground + inpainted background
foreground Ino-aniti inpeintedd background
irput image
correct
detail
backeround patterne mismmtich
larger
detail: repested
patterne
smaller
(1)
(d)
(6)
Figure 1: Exploiting inpainting and movability. (a) Input image. (b) Examples of predicted segmenta- tion masks: correct (top), larger (middle) and smaller (bottom). (c) Inpainted backgrounds the three corresponding cases. (d) Composite image obtained by shifting the foreground object in the three cases. (c) It can be observed that when the mask is incorrect (it includes parts of the background or it does include all of the background), the background inpainting combined with shifting reveals repeated patterns and mismatching background texture, when compared to the original input image or composite images obtained without shifting.
reconstruction error as a supervision signal, we rely on the detection of artifacts generated through shifting, which find to provide stronger guidance.
Suppose that, given a single image (Figure [ (a)), we predict a segmentation mask (one of the 3 cases in Figure (b)). With the mask (Figure (c)).. Then, we can also extract the foreground object, randomly shift it locally, and paste it top of the inpainted background (Figure (d)). When the mask does not accurately follow the outline of a foreground object (e.g., as in the middle and bottom rows in Figure D we can duplication artifacts (of the foreground of the background). We exploit these artifacts as supervision signal to detect the correct segmentation mask. As inpainter we use a publicly available Masked AutoEne (MAE) 20 trained with an adversarial loss [' segmenter
remove the object and inpaint the background
a pre-trained SSL ViT as backbone (e.g.. DINO 4 or the MAE encoder (2I)). We then train neural network head based on an upsampling Convolutional Neural Network (CNN). Following (12), we also further refine the segmenter by training a second segmentation network (SelfMask (2)) with supervision from useudo-mask generated by our trained segmenter. Even without these further refinements MOVE shows remarkable performance on wide range of datasets and tasks. In particular, in unsupervised single object discovery on VOC07, VOC12 and COCO20K improves the SotA CorLoc between 6. % and 9.3%, and in unsupervised class agnostic object detection on COCOva12017 it improves the Ao by 6.8% (a relative improvement of 56%), the AP75 by 2.3% (relative 55%) and the AP by 2.7% (relative 49%).
2 Method
Our objective is to train a segmenter to map a real image x E Hx with the height and W the width of the image, realistic image for any small shifts of the foreground. The mask allows to cut out the foreground from and to move it arbitrarily by some 8 E 2 shift (see Figure 2 top-left). However, when the shifted foreground is copied back onto the background, missing pixels remain exposed. Thus, we
mask
IR xW of the foreground, such that can synthesize

---

Compose
Fake
Real
Gredlent CORR
shiited AE image
shitted mask
Inpainted background
composite imagge w/ shift
AE image
Compose
Compase
fi
Discriminato
AE image
copy- paste fakke
composite Image w/o shift
inpainted background
AE image
image
Fake Real
Figure 2: Synthetic and real images used to learn how to segment foreground objects. We obtain the predicted mask and inpainted background from our segmenter and MAE respectively. We train the segmenter an adversarial manner so that the composite image with shifted foreground (left, top row) looks real. A discriminator is trained to distinguish two types of real (right) from two types of fake (left) images. The fake images consist the composite image with a shift and copy-paste image, obtained by placing the shifted foreground real images consists of composite images without a shift and the real images. The real images are first autoencoded with MAE to match the artifacts of the inpainted background.
top of the input image. The set of
inpaint the background with a frozen pre-trained MAE and obtain b (see Figure Moreover, there is a difference between the texture of i, which is generated from a neural network, and the texture of the cut out foreground from x, which is real image. To ensure more similarity between these two textures, we synthesize 28 by extracting the foreground from the autoencoding (AE) of the input image z shifted by 6, which we call is, and by pasting it onto the background
We enforce the realism of the synthesized images ãt by using adversarial training, i.e., by training the segmenter against a discriminator that distinguishes two sets of real (Figure 2 right hand side) from two sets of fake images (Figure left hand side). The synthetic real image fa-o is obtained by composing zero-shifted foreground with the inpainted background; the second real image ö is instead simply the AE of The two fake images are obtained by composing d-shifted foreground with either the inpainted background b or á, and obtain iss and ãs respectively.
We introduce all the above synthetic images so that the discriminator pays attention only to artifacts due incorrect masks from the segmenter. Ideally, the segmenter should generate masks such that the fake image it looks as realistic as ã for any small ò. However, the discriminator might distinguish these two images because of the background inpainting artifacts and not because of the artifacts due to an incorrect segmentation (which are exposed by random shifts). To avoid this undesired behavior, we also introduce the real image -5=0- This image has no segmentation artifacts for any mask, but has the same background inpanting artifacts as the fake images (although there is no shift in 25=0, the background inpainting creates artifacts beyond the boundaries the segmentation mask). Finally, to guide the discriminator to detect repeated patterns (as those caused by incorrect masks, see Figure [. we also add a fake image Gg. where the background has the original foreground.
The segmenter is trained only through the backpropagation from å.. The details of the segmentation network, the inpainting network and the adversarial training explained in the following sections.
2.1 Segmenter
Following the recent trend of methods for unsupervised object segmentation 7-12 22), we build our method on top of SSL features, and, in particular, DINO 4 or MAE (21 features. Thus, as backbone, we adopt the Vision Transformer (ViT) architecture 23 Following the notation in [10] we split an image æ E I XWWXXX x3 in tiles size P P pixels, for a total of N = HWWPP p2 tiles (and we assume that H and W are such that H/ and W/P are integers). Each tile is then mapped
"The MAE 21 we use is based on VIT architecture and has been pre-trained in adversarial fashion (as opposed to the standard training with MSE loss) output realistic-looking details

---

ting
MAE (inpainter) encodder
impul image
CLS
masking
0000
SSL enooder
MAF (inpaitter) decoder
Segmenter
prodicted
mask
union wf shifted
pooling
inpainted background
Figure 3: (Left) The segmenter is built on top of SSL features from a frozen encoder. To define the inpainting region for the background, the predicted mask is shifted and combined with the unshifted mask (bottom left). For better visualization purposes we highlight the edge the shifted mask, but this does not appear in the actual union of the masks. This mask union is then downsampled to the size of the tile grid via max pooling and denoted m. (Right) The inpainter based on a frozen MAE. First, it takes all the tiles from the input image and feeds them to the MAE encoder. Second, it takes a convex combination between the encoder embeddings and the MSK learned embedding (but now frozen), where the convex combination coefficients are based on the downsampled mask vit. Finally, this combination is fed to the MAE decoder to generate the inpainted background.
through a trainable linear layer to an embedding of size and an additional CLS token included in the input set (see Figure left).
The segmenter network a CNN that takes SSL features as input (e.g., from pre-trained DINO or MAE encoder), upsamples them and then outputs a mask for the original input image. The final output is generated by using a sigmoid ensure that the mask values are always between and 1. We also ensure minimum size of the support of the predicted mask by using
Lyin D max
(iiin Ep
(+) [p HW .00
where n is the number of images the training dataset, (0) the predicted mask from image (00) p is pixel location within the image domain, and Omi is threshold for the minimum mask coverage percentage respectively (in the range [0, 1], where 0 implies that the mask is empty and 1 implies that the mask covers the whole image). Since masks should only take binary values to clearly indicate segment, we use a loss that encourages (*) to take either or values
Cpin 2 Σ HW Σp min {m ( [p), (p)
2.2 Differentiable inpainting
The main task of MOVE is predict a segmentation mask that can be used synthesize realistic image, where the foreground object is shifted on top of the inpainted background (see T top and Figure top left). [igureB]show how we use the predicted high resolution mask for inpainting with MAE. Since MAE performs inpainting by masking or retaining entire patches of P' P' pixels, is necessary to also split the segmentation mask into a grid of tiles of P' pixels and to map each tile to single scalar between () and 1. We do that by using max pooling operation within each tile and obtain low-res mask m, such that mask. To regularize the predicted mask the mask losses Lmin. Loin are also computed max
does not contain any part of the predicted

---

pool m and average pool downsampled masks (at scale 1/P" of the original image resolution; for more details see the supplementary material), Then, we feed the entire set of image tiles the MAE encoder and obtain embeddings f1...., Ex. Next, for ...... N, we compute the convex combination between the embeddings E, and the learned MSK (masked) token from MAE by using the low res mask ri as f = mi[i] EvSK + (1 - 1-mmm)] ( E. Finally, we feed the new embeddings & in the MAE decoder and reassemble the output tiles back into the inpainted background image b (see Figure bottom-right). Notice that we feed all the tiles as input to obtain a differentiable mapping that we can backpropagate on. Interestingly, the MAE encoder, the embeddings E do not store significant information about their neighbors (see the supplementary material). This is in contrast the typical use of MAE, where only the subset of "visible" tiles is fed as input to the encoder. However, such tile selection operation would make the inpainting not differentiable.
found that when tile is masked at the input of
2.3 Adversarial training
Figure 2 shows how we create the images used in the adversarial training. First, we mask the input image with the predicted mask and compose with the inpainted background image, obtaining
28 [p] = ms(p)Hp + 6] (1 (p(p)))(p).
(3)
where ms[p] m jp + +0. 0).8 E [-AW, AW] |-AH, AH] is 2D shift, with A the maximum shift range (relative to the image size). To make the inpainting artifacts the no-shift composite image #5=0 more comparable to those in the shifted composite image, we define the background inpainting region as the union between the predicted mask and its shifted version (see Figure 3.. Thus,
m maxpool p (1 m) о (1 ms))-
To improve the discriminator's s ability to focus on repeated patterns artifacts, we additionally create fake images with a predicted shifted foreground pasted on top of the autoencoded image, obtaining ãg- = 3g @ ma + & ④ ms).
The adversarial loss for the discriminator can be written
LaivD = EE,, min(0. D(zn) - 1} Egs - min - 1}
(5)
and samples for [ ji=1...,.. with uniform random samples 8 42([-AW, AW] x [-AH, A and E denotes the expectation. To speed up the convergence, we also use the projected discriminator method (24). For the segmenter, we use instead the standard loss computed on the composite shifted images
where samples for "real" images æ are the set (=((0))m1
synthetic images zs are the set )==1.... { U1
Lavss = - E2g
(6)
Finally, with Amins Asin nonnegative hyperparameters, our optimization is the adversarial minimization
St = arg min Ladis + Amin Lmii + Abin Lrin
3 subject to D* = arg min LaIDD
(8)
3
Implementation
Except for the ablation studies, in all our experiments we use self-supervised DINO 4 ViT-S/8 transformer pre-trained on ImageNet (25) as an SSL feature extractor. We take the output of the penultimate transformer block of DINO as the feature tokens with ! 8 and feed them to the segmenter. Our segmenter is small upsampling convolutional neural network. It assembles the DINO features into grid of size H/P W/P and processes them with upsampling blocks, so that the output matches the input image resolution. Each upsampling block first performs 2 x 2 nearest upsampling, followed by a x 3 convolutional layer with padding, batch normalization (26) and a LeakyReL.U activation function (see the supplementary material for details). We add an additional block without upsampling and a linear projection to I channel, representing the mask. Our inpainting network is a VIT-L/16 transformer pre-trained on ImageNet Masked Autoencoder
5

---

Table 1: Comparison to the unsupervised saliency detection methods on 3 benchmarks
DUT-OMRON
DUTS- TE
30
ECSSD
Model
Acc
IoU
max F2
Acc
IoU
maxFa
Ace
IoU
max
HS
32
843 .433
.561
8266
.369
.504
.847
.508
.673
wCrr
838 .416
.541
.835
.392
.522
.862
.517
.684
WSC
.865 .387
.523
.862
.384
.528
.852
.498
.683
DeepUSPS
35
.779 .305
.414
.773
.305
.425
.795
.440
.584
SeltMask pseudo
12
811 .403
.845
.466
.893
.646
BigBiGAN 36
856 .453
.549
.888
.498
.899
.672
.782
E-BigBiGAN 36
860 464
.563
882
.511
.624
,906
.684
.797
Melas-Kyriarz et al.
883 .509
.893
.528
.155
.713
LOST [10
.797 .410
.473
.518 .871
.611
.895
.654
.758
Deep Spectral
22
.567
.514
.733
TokenCut 1
880 .533
.600
.903
.576
.672
.918
.712
.803
FreeSOLO
.999 .560
.684
.924
.613
.750
.917
.703
.858
MOVE (Ours)
.23 .615
.712
.950
.713
.815
.954
.830
.916
LOST 10 Bilateral
.888 ,489
.578
.887
.572
.697
.966
.723
.837
TokenCut I|] Bilateral
.977 .618
.697
914
.624
.755
.934
.772
.874
MOVE (Ours) Bilateral
.911 .636
.734
.951
.687
.821
.953
.801
.966
SelfMask pscudo"
12
923 .609
.733
.938
.648
.789
.943
.779
.894
SelfMask on pscudo"
12
Bilateral
.939 .677
.774
.949
.694
.819
.951
.803
.911
SelfMask on MOVE (Ours)
.933 .666
.756
.954
.728
.829
.956
.835
.921
SelfMask on MOVE (Ours) + Bilateral
.937
.665
.766
.952
.687
.827
.952
.800
.917
"We found that SelfMask's max F3 metrie computed with optimal threshold for cach image instead of the entire dataset as in other methods; we re-evaluated their model for fair comparison
(MAE) 21 with an adversarial loss to increase the details of the reconstructed images. For the discriminator we use the Projected Discriminator [24] in its standard setting, but we only use color differentiable augmentation. For the training we use random resized crops of size 224 with scale in range (0.9, 1) and aspect ratio (3/ (3/4, 4/3). We set the minimum mask area 6 = 0.05, the minimum loss coefficient Amin 100 and we linearly ramp up the binarization loss coefficient Apia from 0 to 12.5 over the first 2500 segmenter iterations. We use the shift range A = 1/s. We train the segmenter by alternatively minimizing the discriminator loss and the segmenter losses. Both are trained with learning rate of 0.0002 and an Adam (27 optimizer with betas (0. 0.99) for the discriminator and (0.9, 0.95) for the segmenter. We implemented our experiments in PyTorch (8]. We train our model for 80 epochs with a batch size of 32 on single NVIDIA GeForce 3090Ti GPU with 24GB of memory,
4 Experiments
4.1 Unsupervised saliency segmentation
Datasets. We train our main model using the train split of the DUTS dataset (DUTS-TR) 30 containing 10,553 images of scenes and objects of varying sizes and appearances. We emphasize that we only use the images without the corresponding ground truth. For comparison, we evaluate our approach on three saliency detection datasets: the test set of DUTS (5,019 images), DUT-OMRON 22 (5, 168 images) and ECSSD 31 (1,000 images). We report three standard metrics: pixel (rocisionn Recall for mask accuracy (Acc), intersection over union (IoU), max Fg,, where F3 Preciisoon Recall
0.3; the max Fa is the score for the single optimal threshold whole dataset. Additionally, we report the loU on the test split (38 of CUB-200-2011 (CUB-Birds) 39 dataset.
We train our segmenter in an adversarial manner as specified sections and 3 and evaluate it on the test datasets. We compare with other methods in Table [ Note that without any type of post-processing of our predicted masks, we surpass all other methods by significant margin. We also follow [2] and further refine our masks with bilateral solver [40].
Evaluation.

---

Figure 4: Qualitative evaluation of MOVE on ECSSD, DUTS-TE and DUT-OMRON, First row: input image; second row: MOVE; third row: SelfMask on MOVE; last row: ground truth. Best viewed in color. For more examples and gray scale version see the supplementary material.
Since the bilateral solver only marginally improves or even decreases the quality of our segmentation, conclude that predicted masks are already very accurate. Using the bilateral solver might also inadvertently discard correct, but fragmented segmentations, as we show in the supplementary material, Next, extract the predicted unsupervised masks from the DUTS-TR dataset and use them as pseudo ground wouund-truth truth to train class-agnostic segmenter. We the same architecture (a MaskFormer [41]) and training scheme as SelfMask [12). We then evaluate again on the saliency prediction datasets. Without additional pre- vre-processing our method surpasses or is on par with the SotA across all metrics and datasets. While additional processing with the bilateral solver seems to benefit SelfMask [12], mostly hurts performance our method. Figure 4 shows qualitative results of our method. Finally, we evaluate our method on the test set of CUB-Birds dataset. Additionally, train model on the train split of CUB- Birds dataset and run the same evaluation. We present the comparison with other methods in Table and show that achieve SotA performance.
Table 2: Comparison of unsuper-
vised segmentation methods on the CUB-200-201 test set. MOVE* was trained on the CUB-200-22 train set, while MOVE was trained on DUTS-TR
Method
IoU
PerturbGAN
15
0.360
ReDO 38
0.426
IEM 18
0.551
Mclas- L Kyriazi
37
0.664
Voynov (3
0.683
Voynov- 36
0.710
Deep Spectral
22
0.769
MOVE
0.814
MOVE
0.858
4.2 Single-object discovery
Datasets. We evaluate trained model (see section 4.1 on 3 typical single-object discovery
benchmarks: the train split of COCO20K [42 43 and the trainval splits of VOCO7 [44] and VOC12 45). Following 110 [1] 43 50)
report the Correct Localization metric (CorLoc), i.e., the percentage of images, where the loU > 0.5 of a predicted single bounding box with at least one of the ground truth ones.
Evaluation. Since our method tends to produce a single segmentation mask for multiple objects in the scene, separate the objects by detecting connected components via OpenCV [5T]. We then convert the separate masks to bounding boxes and choose the biggest one as our prediction for the given image. In Table 3 we compare MOVE with other unsupervised methods and we show that just by using processed masks from our method we achieve SotA results on all three datasets, outperforming even methods that used their bounding boxes to train a Class Agnostic Detector (CAD). We show qualitative results for object detection in Figure 5 We also follow the practice of [IO and use our predicted bounding boxes as pseudo-ge -ooooo-roud truth for training the CAD on each of the evaluation datasets. To train the detector, we use either the largest or all the bounding boxes (Multi) that we obtained from the connected components analysis and after filtering those that have an area smaller than 1% of the image. For the evaluation we take the bounding box with the highest prediction confidence, as done 10 IT]. We use the exact same architecture and training scheme competitors for fair comparison. Training with single bounding box improves the
7

---

Table 3: Comparisons for unsupervised single object discovery. We compare MOVE to SotA object
discovery methods on VOCO7 +4 VOC12 [45] and COCO20K 42 datasets. Models are
evaluated with the CorLoc metric. +CAD indicates training second stage class-agnostic detector with unsupervised "pseudo-boxes labels. z) indicates an improvement of over prior sota
Method
VOCO7
44
VOC12 4S
COCO20K 22 43
Selective Search T0
52
18.8
20.9
16.0
EdgeBoxes 10
5
31.1
31.6
28.8
Kim et
[0
46.4
35.1
Zhange et al. 10
55
46.2
50.5
34.8
DDT+ 10 56
50.2
53.1
38.2
iOSD
10
43
54.5
55.3
48.5
LOD
50
53.6
55.1
48.5
DINO-seg
10
45.8
46.2
42,1
FreeSOLO
56.1
56.7
52.8
LOST 1T0
61.9
64.0
50.7
Deep Spectral
222
62.7
66.4
52.2
TokenCut 11
72.1
58.8
MOVE (Ours)
76.0 (1 7.2)
78,8 (1 6.7)
66.6 (1 7.8)
LOD + CAD 10
56.3
61.6
52,7
TOSD + CAD 10
58.3
62.3
53.0
LOST + CAD T0
65.7
70.4
57.5
TokenCut CAD 11
71.4
75.3
62.6
MOVE (Ours) + CAD
77.1
80.3
69.1
MOVE (Ours) Multi CAD
77.5 (1 6.1)
81.5 (1 6.2)
71.9 (1 9.3)
Figure 5: Qualitative evaluation of object detection MOVE on VOC07, VOC12 and COCO20k. Red is the ground truth, ycllow is our prediction. For more examples see the supplementary material.
performance of our method, while training with multiple ones gives it significant additional boost.
Unsupervised class-agnostic object detec- tion. We evaluate our unsupervised object detection model trained on COCO20K with CAD post-training and compare with SotA unsupervised class-agnosiee object detec- tion. In Table 4 we evaluate MOVE on COCOva12017 and report Average Precision (AP) and Average Recall (AR), as in [8]. MOVE yields remarkable relative improve- ment over the AP SotA of 50% on average.
Table 4: Unsupervised class-agnostic object detection on MS COCO va12017. Compared results taken directly from FreeSOLO [
Method
APsp
AP
AP
AR:
AR
AR
Sel. Search 52
0.5
0.1
0.2
0.2
1.5
10.9
DETReg [57]
3.1
0.6
1.0
0.6
3.6
12.7
FreeSOLO g
12.2
4.2
5.5
4.6
11.4
15.3
MOVE (Ours)
19.0
6.5
8.2
5.7
13.6
15.9
4.3 Ablation study
perform ablation experiments on the validation split (500 images) of HKU-IS 588 validate the relative importance of the components of our segmentation approach. For the ablation we train each

---

model for 80 epochs on DUTS-TR. We report the IoU in Ta Our baseline model trained with 3 different seeds gives a mean IoU 0.818 with std 0.008. Thus we only report results for single run in all experiments.
Mask losses. We validate the importance of the mask losses: minimum mask area, binarization and losses on downsampled max-pooled and avg-pooled masks. We find that the minimum area loss is necessary for method to work, otherwise there is no incentive to produce anything other than empty masks. Removing the binarization loss mask losses on the downsam- pled masks makes the masks noisier, which negatively affects the results.
Table 5: Ablation study. Models evaluated on HKU-IS-val
Setting
IoU
Baseline (shift 2/16) no min. mask no binarization loss no pooled mask losses
0.819
0.000
0.774
0.811
no shift
0.000
Shift range. We evaluate different ranges of the ran- dom shift 6. A small range A 1/16 makes it more challenging for the discriminator to detect inconsisten- cies at the border of objects. Larger shifts may cause ob- jects to go out of the image boundaries (A = 3/16, 4/16) and thus reduce the feedback the object boundary to the segmenter. For A 0 (no-shift) the only possible discriminator inputs are composed images without shift as fake and autoencoded images as real. There is no incentive to produce any meaningful masks in this case.
shift 1/16
0.751
shift 5/16
0.799
shift 4/16
0.704
disc. fake inputs: composed disc. real inputs: + comp. w/o shift
0.789
0.740
disc. real inputs: comp. w/o shift disc. real inputs: Fae non-diff' inpainter MSE MAE MAE feature extractor ImageNet100 dataset
0.031
0,000
0.314
0.817
0.783
0.815
Discriminator inputs. In our baseline model, we feed
both composed images with no-shift and real images autoencoded with MAE as real samples and composed images with shift and autoencoded images with copy-pasting of predicted masked object as fake samples for the discriminator training. We test the case DISC. REAL + COMP. w/o SHIFT, where we feed to the discriminator real images without autoencoding. In this case, the discriminator can detect the artifacts of MAE instead of focusing on inconsistencies resulting from an incorrect mask. In DISC. REAL Zae we only feed the autoencoded images as real. Here, the discriminator focus on the mismatch from the inpainting artifacts and encourages the segmenter to output empty masks, where no inpainting is done. If we only feed the composite non-shifted images (DISC. REAL COMP w/o SHIFT), the artifacts resulting from an incorrect masks cannot be fixed, because there no reference of what the real images look like. In DISC. FAKE INPUTS: COMPOSED we only feed the composed image as fake to the discriminator and omit the real image with a copy-pasted predicted masked object, which slightly degrades the performance.
Non-differentiable inpainter. We evaluate the use of hard thresholded downsampled masks as input to the background inpainter. In this case the only feedback for the masks comes from the composition of the images. We find to be insufficient for the segmenter to learn any meaningful masks.
Inpainter model. We substitute the MAE trained with GAN loss with a MAE that was trained only to reconstruct missing patches with a Mean Squared Error (MSE) loss. Since this model was trained to only reconstruct the missing patches and not the entire image, we construct the in- painted background by composing the inpainted part with the real image: map = u (m:); i = x @ (1 - mp) + i @ mup-- Consequently, we do not use autoencoding when creating the discriminator inputs. We find this model perform competitively.
Feature extractor. We train the model using the features provided by MAE encoder instead of separate DINO model. In this case we adapted the segmenter architecture and added one more upsampling block, since MAE takes patches of size P = 16 (instead DINO has P 8). We find that with these features we are able to train competitive segmenter.
ImageNet100 Net100 dataset. We train our model on the ImageNet100 dataset [59] with 131,689 images from 100 randomly selected ImageNet (25] classes Since this dataset is much bigger than DUTS-TR, we adapt our segmenter by adding an additional convolutional layer in each upsampling block (see section 3) and train the model for 8 epochs. The results are comparable to the DUTS-TR dataset.
9

---

5
Prior Work
In the past decade, research on object segmentation and detection has seen remarkable progress when full supervision is available [60]
62 To limit the cost of annotation several methods explored
different forms of weak supervision 663 64 or ways to avoid labeling altogether [33 14 65, [6].
MOVE falls the latter category. Therefore, we focus our review of prior work on unsupervised methods for object segmentation and the related task of object detection.
Unsupervised Object Detection and Category Discovery. Unsupervised object detection and category discovery are extremely challenging tasks that have recently seen a surge of efforts [57] 67 68] thanks to the capabilities of modern deep learning models. Recently, features based on deep learning have shown significant progress in object detection 577 even with just some noisy (unsupervised) guidance [52]. More in general, one limitation of unsupervised object detection is that it only provides a coarse localization of the objects. As have shown with MOVE, it is possible to obtain much more information without supervision.
Unsupervised Object Segmentation. Object segmentation can be formulated as pixel-wise image 69 70 or through the generation of layered models from which a segmenter is trained byproduct 8 35 72.. The use of SSL. features has spawned several methods with significant performance on real datasets, which we discuss in the following paragraphs.
partitioning task [65] (66]
SSL-Based Methods. Due the success of SSL methods and the emergence of segmentation capabilities, several recent methods for unsupervised object segmentation have been built on top of SSL features. In particular, SelfMask [12] proposes a clustering approach that can use multiple SSL features and evaluates all possible combinations of DINO A SWAV (73 and MOCOV2 77). They find that combining features from all three SSL methods yields the best results for segmentation. FreeSOLO [8] instead finds that DenseCL features
[6] work best. More in general, some methods use
a weak (unsupervised) guidance and losses robust to the coarse pseudo-labels (8, but the majority is
based on directly clustering SSL. features 77 1 [2 75 In contrast to these methods, we show
that movability can provide robust supervision signal.
Generative Methods. A wide range of methods also exploits generative models to create layered image representations [ 17 20 76H77). A general scheme is to train a network to generate a background, foreground and its mask. These components can then be combined to generate an image and then, in a second stage, one can train a segmenter that learns to map a synthetic image to its corresponding foreground mask. In alternative, the segmenter could be built during the training of the generative model as a byproduct. Some methods rely on the assumption that a dataset of only backgrounds is available [14, [0). The use of shifts to define segments has also been used before 13 15 16]. However, MOVE does not require the training of a generative model, which can be a challenge on its own.
6 Limitations and Societal Impact
L Limitations. As mentioned in the introduction, movability alone may not suffice in identifying an object unambiguously. In fact, the method can segment any combination of multiple objects. To address this we use post-processing algorithm to find connected components, but there is no guarantee that all objects have been segmented. Another issue is that shifts would not expose artifacts when the background is uniform (e.g., looking at the sky, underwater, with macrophotography). Societal Impact. An important aspect that is relevant to unsupervised learning methods in general is the potential become biased if the training datasets are unbalanced. This clearly has a potentially negative impact on the segmentation of categories that are underrepresented and thus this work should be integrated with mechanisms take the dataset imbalance into account.
7 Conclusions
We have introduced MOVE, a novel self self-supervised method for object segmentation that exploits the synthesis of images where objects are randomly shifted. MOVE improves the SotA in object saliency segmentation, unsupervised single object discovery, and unsupervised class agnostic object detection by significant margins. Our ablations show that movability is strong supervision signal that can be robustly exploited as a pseudo-task for self-supervised object segmentation. We believe that our approach can be further scaled by exploring different architectures and larger datasets.
10

---

References
[1] Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affor- dance for direct perception in autonomous driving. In Proceedings of the IEEE international conference on computer vision, pages 2722-27322 2015.
[2] Mang Tik Chiu, Xingqian Xu, Yunchao Wei, Zilong Huang, Alexander G. Schwing, Robert Brunner, Hrant Khachatrian, Hovnatan Karapetyan, Ivan Dozier, Greg Rose, David Wilson, Adrian Tudor, Naira Hovakimyan, Thomas S. Huang, and Honghui Shi. Agriculture-vision: A large aerial image database for agricultural pattern analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[3] Erik Smistad, Thomas Falch, Mohammadmehdi Bozorgi, Anne C Elster, and Frank Lindseth. Medical image segmentation on gpus-a comprehensive review. Medical image analysis, 20(1): 1-18, 2015.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[5] Xinlei Chen", Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02053 2021.
[6] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[7] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.058144 2021.
[8] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. FreeSOLO: Learning to segment objects without annotations. arXiv preprint 22X.:22222.12811,, 2022.
[9] Yin Zhaoyun, Wang Pichao, Wang Fan, Xu Xianzhe, Zhang Hanling, Li Hao, and Jin Rong. Transfgu: A top-down approach to fine-grained unsupervised semantic segmentation. arXiv preprint arXiv:2112.01551, 2021.
[10] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self self-supervised transformers and no labels. November 2021.
[11] Yangtao Wang, Xi Shen, Shell Hu, Yuan Yuan, James Crowley, and Dominique Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. arXiv preprint arXiv:2202.11539 2022.
[12] Gyungin Shin, Samuel Albanie, and Weidi Wie. Unsupervised salient object detection with spec- tral cluster voting. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), June 2022.
[13] Tal Remez, Jonathan Huang, and Matthew Brown. Learning to segment via cut-and-paste. In The European Conference on Computer Vision (ECCV), September 2018.
[14] Pavel Ostyakov, Roman Suvorov, Elizaveta Logacheva, Oleg Khomenko, and Sergey I. Nikolenko. SEIGAN: towards compositional image generation by simultancously learning to segment, enhance, and inpaint. CoRR, abs/1811.076600 2018. 1811.07630
URL http:://///a org/orrg..................o.
[15] Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. In Advances in Neural Information Processing Systems, pages 7256-7266, 2019.
[16] Relja Arandjelović and Andrew Zisserman. Object discovery with a copy-pasting gan, arXiv preprint arXiv: 1905. 2019.

---

[17] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[18] Pedro Savarese, Sumnie SY Kim, Michael Maire, Greg Shakhnarovich, and David McAllester. Information-theeortic segmentation by inpainting error maximization. arXiv preprint arXiv:2012.07287, 2020.
[19] Isinsu Katircioglu, Helge Rhodin, Victor Constantin, Jörg Spöri,, Mathieu Salzmann, and Pascal Fua. Self-supervised human detection and segmentation via background inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
[20] Jianwei Yang. Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive generative adversarial networks for image generation. ICLR, 2017.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2 1 2021.
[22] Luke Melas Pyriaa,,riazz,, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral meth- ods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.1 1929, 2020.
[24] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[25] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[26] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pages 448-456 PMLR, 2015.
[27] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 2025--0035 Curran Associates, Inc., 2019. 9015 2015-pytorch-- wwtppporroo---nnrrraareeeeeeeeee cna-iiperative--ty//e hwwp:///-e-ip-pprroooooooe-ee-ee ott.a//e.....t..............o.o. -ean--i--rannnnnn-brrrrr pdf
[29] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph- manifold ranking. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3166-3173. IEEE, 2013.
[30] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 136-145, 2017.
[31] Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierarchical image saliency detection on extended essd. IEFE transactions on pattern analysis and machine intelligence, 38(4):717-729 2015.
12

---

[32] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia. Hierarchical saliency detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1155-1162, 2013.
[33] W. Zhu, Liang, Y. Wei, and Sun. Saliency optimization from robust background detection. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 28 181--2221,
2014.
[34] Nianyi Li, Bilin Sun, and Jingyi Yu. A weighted sparse coding framework for saliency detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5216-5223, 2015.
[35] Tam Nguyen, M Dax, C Mummadi, N Ngo, Z Lou, and Thomas Brox. Deepusps: deep robust unsupervised saliency prediction via self-supervision. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[36] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large-scale generative models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Muchine Learning Research, pages 10596-10606. PMLR, 18-24 Jul 2021. URL https
//proceedings.oo wtw.s/ppee///ee wttppp/eeee///eeeeeepee htt///////////oo///// html
[37] Luke Melas-Kyriazi Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an unsuper- vised image segmenter in each of your deep generative models. 2021.
[38] Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019, URL
www.eeeeriiiiiomc pee.ripp.ccc/ppppr///////11e .dff
[39] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD 20rds-200-2011 Dataset. Technical Report CNS-TR-201 TTT-2011-000 2001, California Institute of Technology. 2011.
[40] Jonathan Barron and Ben Poole. The fast bilateral solver. European Conference on
Computer Vision, pages 617-632. Springer.
[41] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances Neural Information Processing Systems, 34, 2021.
[42] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2014. URL http:://rx//.ii.....// org/ abs/ 1405. 0312
[43] Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In European Conference on Computer Vision, pages 779-795. Springer, 2020.
[44] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal. htmll,
[45] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal. html,
[46] Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 1201-1210, 2015.
[47] Thomas Deselaers, Bogdan Alexe, and Vittorio Ferrari. Localizing objects while learning their appearance. In European conference on computer vision, pages 452 466. Springer, 2010.
[48] Parthipan Siva, Chris Russell, Tao Xiang, and Lourdes Agapito. Looking beyond the image: Unsupervised learning for object saliency and detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3238-3244 2013.
13

---

[49] Huy V Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce. Unsupervised image matching and object discovery as optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8287-8296, 2019.
[50] Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsupervised object discovery. Advances in Neural Information Processing Systems, 34, 2021.
[51] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
[52] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International jourmaa of computer vision, 104(2):14-171, 2013.
[53] C. Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 391-405, Cham, 2014. Springer International Publishing. ISBN 978-3-319- 10602-1.
[54] Gunhee Kim and Antonio Torralha. Unsupervised detection of regions of interest using iterative link analysis. Advances in neural information processing systems, 22, 2009.
[55] Runsheng Zhang, Yaping Huang, Mengyang Pu, Jian Zhang. Qingji Guan, Qi Zou, and Haibin Ling. Object discovery from a single unlabeled image by mining frequent itemsets with multi-scale features. IEEE Transactions on Image Processing, 29:8606-8621, 2020.
[56] Xiu-Shen Wei, Chen-Lin Zhang, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou. Unsupervised object discovery and co-localization by deep descriptor transformation. Pattern Recognition, 88:1 13-126, 2019.
[57] Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region priors for object detection, 2021.
[58] G. Li and Y. Yu. Visual saliency based on multiscale deep features. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5455-5463, June 2015.
[59] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer, 2020.
[60] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988 2017.
[61] Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[62] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[63] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[64] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.00 077550000.......... 2019.
[65] Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. CoRR, abs/1807. abs//880.00653. 2018. http::/ttt//axxi //rxiv org ht////bs/1107 111110677770666363
(66] Asako Kanezaki. Unsupervised image segmentation by backpropagation. Proceedings of
IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2018.
14

---

[67] Sai Saketh Rambhatla, Rama Chellappa, and Abhinav Shrivastava. The pursuit of knowl- edge: Discovering and localizing novel categories using dual memory. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9153-9163, 2021.
[68] Jiyang Zheng, Weihao Li, Jie Hong. Lars Petersson, and Nick Barnes. Towards open-set object detection and discovery. arXiv preprint arXiv:2204.05660 2022.
[69] Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmen- tation. In European Conference Computer Vision, pages 142-158. Springer, 2020.
[70] Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. CoRR, abs/1 11111711.0080 1.08006, 2017.
[71] Jing Zhang. Tong Zhang, Yuchao Dai, Mehrtash Harandi, and Richard Hartley. Deep unsuper- vised saliency detection: A multiple noisy labeling perspective. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[72] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: 1399, 2019.
[73] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9922 2020.
[74] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[75] Adrian Ziegler and Yuki M. Asano. Self-supervised learning of object parts for semantic seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[76] Sjoerd van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-325, 2020.
[77] Hanock Kwak and Byoung- Zhang. Generating images part by part with composite genera- tive adversarial networks. arXiv preprint FrXXiv:6677.5388 2016.
[78] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 2225-3233. 2016.
[79] Xingzhe He, Bastian Wandt, and Helge Rhodin. Ganseg: Learning segment by unsupervised hierarchical image generation. arXiv preprini arXiv:21 2021.
[80] Yaniv Benny and Lior Wolf. Onegan: Simultancous unsupervised learning of conditional image generation, foreground segmentation, and fine- grained clustering. In European Conference on Computer Vision, pages 514-530. Springer, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
15

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)) [No] We use publicly available datasets. We plan on sharing the code and the instructions at a later time.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Ys]] We trained our baseline model with different seeds and reported mean and std IoU on the ablation dataset. We do not include it for all the experiments due to time and cost constraints.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'reree using/curating? [No] We use datasets publicly available for research
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "u9dj4F0cI41", "reviewer": "Reviewer_tM6J", "summary": "The paper proposes an unsupervised segmentation approach using the \"movability\" as the cue to differentiate between foreground and background. The pretext task is to segment the foreground object, shift it into a new location, and compose the new image with inpainted backgrounds. The approach relies heavily on pretrained models, DINO for the segmentation network, and MAE for the inpainting network. A GAN loss is used to encourage the composed image to be realistic. The method assumes that the segmentation network has to segment the foreground object in order to synthesize realistic images. Encouraging results are reported on saliency detection and unsupervised object discovery.", "questions": "- I think it is inappropriate to use the term \"movability\" to describe such method. It would confuse readers that the method exploit motion cues to segment objects in videos. It is also not correct that any foreground object could be displaced to any location. For example, the monkey could not be displaced to the sky. Also, other background objects or stuffs could be displaced properly. For example, the tree log could be moved left or right. \n\n- In figure 3, the encoded foreground looks identical to that of the input foreground. Why do we have to encode foreground through MAE?\n\n- Also in figure 3, the inpainted background looks unreal. The grassland behind monkey now becomes homogeneous green region. Why is this a meaningful task? The discriminator would only needs to find these inpainting artifacts in order to discriminate between real and fake. No matter predicted mask is, the inpainted results will always have artifacts which may impede GAN learning. \n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The idea of using inpainting to formulate a self-supervised learning pretext task is interesting.\n\n+ The empirical result is encouraging. Notable improvements are obtained against prior works on all benchmarks. \n\nWeaknesses:\n- The method relies heavily on pretrained models, yet without discussing the significance of it. I suspect that the method would not work at all without pretrained DINO & MAE models. The general claim of the method should work even without any pretrained models. I am not convinced about the paper it does not work standalone.\n\n- The writing is poor, especially in the approach section page 4. Description of the method is disorganized. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "O5f9HP3imeg", "reviewer": "Reviewer_9LXw", "summary": "This paper proposes a way to segment objects in unsupervised fashion.  The idea is to define a signal to a segmentation model based on the idea that if the object were properly segmented, then the predicted mask could be used to shift the object (by a small translation within the same image) in such a way that this new shifted image looks realistic as judged by a discriminator.  As part of generating this new shifted image of the object, the authors propose to use MAE (masked auto-encoders) to inpaint the missing region which is uncovered by the simulated shift.\n\nThe paper demonstrates that this proposed model “MOVE” outperforms comparable SOTA models on unsupervised saliency detection and unsupervised object detection benchmarks.\n", "questions": "* The cut+paste citation (Remez et al 2018) is slightly mischaracterized — this paper did not require pasting objects to different images (it was always to the same image).  Related to this, the Remez et al approach had an issue of inferring that shadows were part of an object.  What is the behavior of MOVE when there are shadows?\n* In Section 5, the authors claim that MOVE does not require training of a generative model —- but this seems somewhat strange — is there a clear line that separates a model such as MAE from generative models?\n* How necessary was it to use a DINO pretrained backbone?  Would starting from scratch have worked?\n", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I enjoyed reading this paper — the results are strong and seem to quite significantly outperform the state of the art.  Given the complexity of the approach, getting this level of performance is quite impressive and points to good execution.  Conceptually, the trick that the authors propose to deal with artifacts generated by MAE (specifically they propose to synthesize these artifacts in images where the object is unshifted which forces model to focus on differences other than the artifacts) is also simple and effective.\n\nThe main weakness of the approach is probably the complexity of the model which requires a segmentation model, a separately pre-trained masked auto-encoder and training both in a GAN-like setting, so I suspect that reproducing this model is not easy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9zwlLmOTrt", "reviewer": "Reviewer_tSAq", "summary": "This work introduces a framework for unsupervised image segmentation. The assumption that it exploits is that a slightly shifted foreground segment overlayed on an inpainted background image, whenever the former is meaningful, must look realistic. The work develops and carefully implements this idea using a transformer to generate and inpaint the masks. A discriminator network is tasked to distinguish between the generated images and the real images, where appropriate choices were made to avoid degenerate solutions by augmenting the real image distribution. Elaborate experiments on saliency detection, single object discovery and detection, demonstrate substantially improved accuracy compared to previous state of the art.\n\nOverall, the paper is insightful, original and technically sound. It makes a clear advance in the problem of unsupervised segmentation.\n\n**Post-rebuttal comment.**\nI thank the authors for futher elaborations. Regarding the concern of *R-tM6J*: I agree that using pre-trained features (or even depending on it) is a limitation. However, I do not see it as critical. After all, the pre-training is completely unsupervised, and performing accurate segmentation on the representation learned – especially to the degree demonstrated in this work – is highly non-trivial.\n\nIn short, this is a nice work that deserves to be presented at the conference, which I strongly recommend.\n\n*To the authors:*\n- Please do elaborate on the methodology for tuning the framework, reported in Tab. 1&2, in the revision. This increasingly  becomes a source of confusion in the literature on unsupervised learning, impedes reproducibility and fairness of comparisons.\n- The ablation Table 4 needs some revision. Adding some clarification in the caption, or changing the table layout may help.", "questions": "A few open questions:\n- Since the framework is completely unsupervised, what stands in the way of training it on larger datasets (e.g. ImageNet)? Would it scale?\n- What criteria and data were used to choose hyperparameters?\n- How crucial is the ImageNet self-supervised pre-training of the feature encoder?", "limitations": "Limitations and the societal impact are discussed at appropriate length.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* clearly written;\n* contains a number of technically innovative bits, since implementing this conceptually simple idea is actually prone to degenerate solutions; \n* extensive experimentation with strong results across the board.\n\nA minor weakness:\n\nThe main idea exploited here re-implements the “cut-and-paste” principle from previous studies, but in the context of the latest advances in network architectures (ViT), self-supervised learning (DINO) and autoencoders (MAE). However, this is given appropriate credit in the text (l. 29). Also, the focus on small shifts, in contrast to some previous works, also makes sense geometrically, since perspective distortions can be neglected.\n\n\nA few suggestions:\n\nFig. 2 and 3 are disconnected and show different parts of the pipeline. Having a unified “overview” diagram, that also depicts \n- what images the “fake” and “real” sets comprise (as the input to the discriminator),\n- which network modules remain fixed and which are trained,\nwould help in accompanying the text in Sec. 2, which is is quite dense.\n\nIn Tab. 1-3, it would interesting to see an overview of the computational footprint of those methods (e.g. the backbone / FLOPS) and/or the datasets used for training (e.g. “ImageNet + DUTS-TR”) for this work.\n\nImages in Fig. 4/5 are too small. I’d rather use fewer samples, but in larger size. More images can be always added to the supplemental.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept