R-MONet: Region-Based Unsupervised Scene Decomposition and Representation via Consistency of Object Representations
Authors: Shengxin Qian
Venue:  (2021)
Keywords: unsupervised representation learning
unsupervised scene representation
unsupervised scene decomposition
generative models

Abstract: Decomposing a complex scene into multiple objects is a natural instinct of an intelligent vision system. Recently, the interest in unsupervised scene representation learning emerged and many previous works tackle this by decomposing scenes into object representations either in the form of segmentation masks or position and scale latent variables (i.e. bounding boxes). We observe that these two types of representation both contain object geometric information and should be consistent with each other. Inspired by this observation, we provide an unsupervised generative framework called R-MONet that can generate objects geometric representation in the form of bounding boxes and segmentation masks simultaneously. While bounding boxes can represent the region of interest (ROI) for generating foreground segmentation masks, the foreground segmentation masks can also be used to supervise bounding boxes learning with the Multi-Otsu Thresholding method. Through the experiments on CLEVR and Multi-dSprites datasets, we show that ensuring the consistency of two types of representation can help the model to decompose the scene and learn better object geometric representations.

Under review as a conference paper at ICLR 2021
R-MONET: REGION-BASED UNSUPERVISED SCENE DECOMPOSITION AND REPRESENTATION VIA CONSIS- TENCY OF OBJECT REPRESENTATIONS
Anonymous authors Paper under double-biee blind review
ABSTRACT
Decomposing complex scene into multiple objects is a natural instinct of intelligent vision system. Recently, the interest in unsupervised scene represen- tation learning emerged and many previous works tackle this by decomposing scenes into object representations either in the form of segmentation masks or po- sition and scale latent variables (i.e. bounding boxes). We observe that these two types of representation both contain object geometric information and should be consistent with each other. Inspired by this observation, we provide unsuper- vised generative framework called R-MONet that can generate objects geometric representation in the form of bounding boxes and segmentation masks simultane- ously. While bounding boxes can represent the region of interest (ROI) for gen- erating foreground segmentation masks, the foreground segmentation masks can also be used to supervise bounding boxes learning with the Multi-Oue Thresh- olding method. Through the experiments on CLEVR and Multi-dSprites datasets, we show that ensuring the consistency of two types of representation can help the model to decompose the scene and learn better object geometric representations.
INTRODUCTION
In recent years, supervised object detection and segmentation (He et al. 2017); Ren et al. (2015):
Fan al. (2019); Liao et al. (200I); ); Lin et al. (2017); Ronneberger al. (2015)) have made great progress with the extensive human labels. However, these supervised methods are still unable to take the advantage of massive unlabeled vision data. Unsupervised learning of scene representation starts to become key challenge in computer vision. The breakthrough (Burgess et al. (2019); Greff al. (2019); |Eslami al. (2016): Crawford & Pineau (2019); |Engelcke al. (2020). Greff et al. (2017 Van Steenkiste et al. (2018); ; Pathak et al. (2016); Lin et al. (2020)) in the unsupervised scene decomposition and representation learning proves that a complex visual scene containing many ob- jects can be properly decomposed without human labels. It proves that there is still much useful information that can be discovered in those unlabeled data.
Recent approaches address the unsupervised scene decomposition and representation learning can be categorized into two groups: models which explicitly acquire disentangled position and seale (i.e. bounding boxes) representation of objects (Eslami al. (2066); Crawford & Pineau (2019);
Lin et al.] (2020)) and models implicitly encode objects' geometric representation into segmenta- tion masks entangle with object appearance representations (Burgess et al. (2019): Greff et al.
(2019): Engeleke et al. (2020); Greff et al. (2017); : Van Steenkiste et al. (2018)) ). In the former type of models, the scene is explicitly encoded into the object-oriented spatial encoding and appearance encoding. A decoder will generate the scene with explicitly defined object encoding for representa- tion learning. This type of models cannot use rectangular bounding boxes to fully represent complex objects with flexible morphology. In the other type of models, the scene is decomposed into finite number of object segmentation masks which can better represent complex objects with its pixel-to- pixel alignment. However, this type of models only use segmentation masks as the pixel-wise object mixture weights. They do not utilize the geometric information in the segmentation masks and still entangle object position and appearance representations in the scene generation step. Also, those models tend to decompose the entire scene in the image which does not use the locality benefit of objects.

---

Under review as a conference paper at ICLR 2021
Inspired by the observation that foreground segmentation masks and bounding boxes both contain object geometric information and should be consistent with each other. a method called R-MONet (Region-based Multiple Object Net) is proposed in this paper. R-MONet uses the spirit of MONet (Burgess et al. (2019) and S4Net (Fan et al. (019)) by using a single stage, non-iterative network (spatial attention module) for generating object geometric representations in both bounding boxes and segmentation masks. Then, a variational autoencoder (VAE) (Kingma & Welling (2013)) is used for encoding object appearance representations and regenerating the scene for training. To ensure the consistency between bounding boxes and foreground segmentation masks, the bounding boxes generated from spatial attention module is supervised with the pseudo bounding box generated by Multi-Otsu thresholding method (Liao et al. (2001)) on foreground segmentation masks. More than that, the foreground instance segmentation is only performed in the bounding box area instead of the full image to take advantage of the spatial locality and make scene generation less complex. The contributions of this paper are:
We introduce an effective single stage, non-iterative framework to generate object geomet- ric representations in both bounding boxes and segmentation masks for unsupervised scene decomposition and representation learning.
We propose self-supervised method that can better utilize object geometric information by ensuring the consistency between bounding boxes and foreground segmentation masks. This approach can improve the scene decomposition performance compared with the state- of-art.
We design segmentation head that can preserve global context and prevent coor- dinate misalignment in small feature maps which improves the foreground segmentation performance.
2 RELATED WORKS
There many influential works (Burgess al. (2019) y Greff al. (2019) Eslami al. (2016); Craw- ford & Pineau 2019); Engelcke et al. (2020); Greff et al. (2017); Van Steenkiste et al. (2018); Pathak al. (2016); Lin al. (2020) in unsupervised scene decomposition in recent years. Some models tend to explicitly factor an object representation into spatial and appearance encodings such as 'what', 'where', 'presence', etc. with the help of VAE (Kingma & Welling (2013)). Influential related models include AIR (Eslami et al. (2016)) and its successor SPAIR (Crawford & Pineau 22199). AIR uses the Recurrent Neural Network as the encoder to decompose a complex scene into objects' representation but it suffers from the iteration speed. SPAIR improves its bounding box av- erage precision and running speed by using Convolution Neural Network as the encoder to generate objects' representation in parallel. However, these models have not been tested on photorealistic 3D object dataset and bounding boxes can not fully represent flexible morphology like foreground segmentation masks.
The other type of models tend to decompose each object into its own representation without explicit positional encoding and use segmentation masks mix object reconstruction. Influential models such as MONet (Burgess et al. (211)) which leverages a UNet (Ronneberger et al. (2015) variant as an iterative attention network for segmentation mask generation and Spatial Broadcast Decoder (Watters et al. (2019) for representation learning via scene reconstruction. Spatial Broadcast De- coder replaces deconvolutional network with transform by tiling (broadcast) the latent vector across space, concatenate fixed X- and Y- "="ooodiiate"" channels. This decoder provides better disentangle- ment between positional and non- positioil features in the latent distribution. IODINE (Greff et al. (019)) tackles this problem with its amortized iterative refinement of foreground and background representation. However its iterative refinement process will heavily impact the speed of training and inference. GENESIS (Engelcke et al. (2020)) uses the similar idea as MONet but with different latent encoding in different iterative steps. These models all focus on decomposing the entire scene which does not leverage the spatial locality around each object.
SPACE (Lin et al. (0200) is the closest to our work in spirit. This model leverages the encoder similar to SPAIR process foreground objects in parallel with explicit positional encoding and adapts the segmentation masks for background modeling. However, different from R-MONet, does not use the shared information in both bounding boxes and segmentation masks.
2

---

Under review as a conference paper at ICLR 2021
Tharkig
Superomation
Generation
Maden
Figure I: End-to-End Architecture of R-MONet. (a) A spatial attention network similar to S4Net receives the input image to generate foreground objects' bounding boxes {tx} and segmentation intermediate output {co The scope {8k] represents the portion of the undecomposed scene. With the initial scope 8o 1. the stick-breaking process will transform {au} } into the segmentation masks } (K-1 masks for the foreground objects and mask for the background) and keep E m&
VAE encoder takes the image and (mæ} the inputs to compute representation latents over segmentation and input image {=x}- (b) VAE decoder reconstructs different objects {ēx} and their masks {m+} with {zk} for representation learning. (c) To train the RPN in the inference network, {mx {m ( [-] will be passed into the Multi-Otsu thresholding method to gen- erate the pseudo bounding boxes {?,). The image, {mz) and reconstructed objects {T*) will form VAE ELBO loss. The KL divergence between {m,} } and reconstructed segmentation masks (m²) forces VAE model the segmentation masks distribution in spatial attention module. The details of the loss is in Section 3.5
is worth mentioning that S4Net (Fan al. 2019) provides an end-to-end single shot object detection and instance segmentation framework simplified from Mask R-CNN (He et al.] (2017)) for our spatial attention module. It uses the ROI Masking method inspired by the fact that the segmentation can benefit from the background features surrounding the foreground instance. In this method, an ROI mask generated by enlarging the proposed region on the feature maps. The mask values inside the original proposed region are set to 1. The mask values outside the original proposed region but inside the enlarged proposed region are set to -1. The rest values on the mask are set to O. The regional features are extracted by multiplying the ROI mask and the feature maps. This method is proved to have better segmentation compared with ROI Align (He et al. (2017) in both their experiments and our use case (discussed in Section 4.1
3 METHOD
In this section, the proposed model, R-MONet is described. The R-MONet will decompose the scene into K different components (K-1 1 for the foreground and I component for the background) and each of them is indexed by also limits the max number objects this model can detect. The network architecture of R-MONet is presented in Figure
3.1 INFERENCE
Inference of object geometric representations. A spatial attention network parameterized by ib provides the object geometric representation in both segmentation masks {m²) and foreground object bounding box (4x) with the input image æ. It represented by equation [ At the same
3

---

Under review as a conference paper at ICLR 2021
time, {m is also the mixing probability in spatial Gaussian model. Since this mixing probability is learned by the network conditioned on æ, we refer to this distribution as go (m)/zz).
(EI.k) tte kE = (z)
This spatial attention network a convolutional neural network similar to S4Net (Fan et al. 2019) with small adjustment. In this module, ResNet18 8 (He et al.] (2016) and Feature Pyramid Network (FPN) (Lin et al. (2017)) are used as the backbone. The feature maps are extracted for the following Region Proposal Network (RPN) (Ren et al. (2015). The bounding boxes proposed by RPN will be filtered with Mon-Maximun suppression (NMS) (Neubeck & Van Gool (2006)) Only ROIs with top K-1 prediction scores will be selected for further processing. The ROI Masking (Fan et al. (2019) method then uses the ROI selected by RPN and transforms feature maps from the backbonn to the segmentation head. The segmentation intermediate outputs {ax) J (not segmentation masks) from the segmentation branch will be transformed into the segmentation masks {m&} via the stick-breaking
process (Sethuraman (1994) to make sure k= m& 1 which represents all pixels are explained in the segmentation masks (softmax can also be used). The scope 8% represents the proportion of pixels still unexplained. The details about stick-breaking process is in Appendix A
Inference of object representation latents. In the VAE encoder parameterised by Φ, we use vari- ational inference to get approximate latent posterior (z (x, m&) which is Gaussian distribu- tion conditioned on both input image x' and segmentation masks [m As pointed out in MONet (Burgess et al. (2019)), conditioning on {m&) can provide geometric heuristic information for prob- abilistic inference. Then, we sample from this posterior to infer object representation latents {
3.2 IMPROVED SEGMENTATION HEAD
During our experiments, we test two kinds of segmentation heads. Similar to S4Net, the first one extracts only conv3 layer feature maps for ROI Masking and adapts the similar segmentation head. We refer to as R-MONet(Lite). R-MONet(Lite) can provide decent object geometric represen- tation, but it can not generate sharp edges due to the missing of low level features. Inspired by the ROI Masking and the UNet (Ronneberger et al. (015)). we propose an improved segmentation head which can extract feature maps from the region of interest (ROI) and keep the low level de- tails with the skip connection. We name the proposed framework with the improved segmentation head as R-MONet(UNet) in the following sections The architecture is shown in Appendix A In this new head, feature maps from all layers except the conv4 and conv5 are applied ROI Masking. This design can extract the features in ROI but still preserve global context and prevent coordinate misalignment in small feature maps. Similar to the UNet, the top feature map from conv5 is passed into a small multilayer perceptron (MLP) (John Lu (2010) and the feature maps from each layer have the skip connection to the same level in the segmentation branch. The quantitative comparison between R- R-MONe((L.te)) and R-MONet(UNet) shown in Section 4
3.3 GENERATIVE PROCESS
Assuming there are at most K objects (foreground and background) in the scene, we use K com- ponents in the input image x to represent K object representation latents {zx)- 2g is treated as independent Gaussian random variable with unit prior and the VAE decoder is shared across all components. The generative process is treated as spatial Gaussian mixture model where each mix- ing component represents a single object. This means kth object representation z passed into the VAE decoder parameterised by 0 to generate reconstructed kth object mask m& (referred as po ( =x)) and kth reconstructed object T& (i.e. pixel-wise mean). The kth corresponding mixing probability p(component = k| [m ) = m& (i.e. kth segmentation mask from inference) repre- sents the probability pixels belong to component The input image likelihood given by the mixture
model is:
Po (#x))
mapo zx)
(2)
K-1 m (x;; Tk. )70)
<mxN(x;;
onn) + Σ
4

---

Under review as a conference paper at ICLR 2021
where fixed variance jg for K-1 foreground components and d for the background component. We use the spatial broadcast decoder (Crawford & Pineau (2019)) here for better disentanglement between positional and non-positional features latent distribution.
3.4 REGION BASED SELF-SUPERVISED TRAINING
Since foreground segmentation masks (The first K-1 segmentation masks) and bounding boxes (bboxs) both contain foreground object geometric information and should be consistent with each other, we can choose simple unsupervised thresholding algorithm to separate the pixels the fore- ground masks into several different classes based on the probability of being certain foreground ob- ject. With the separated classes in each component's 's foreground mask, we can generate pseudo posi- tive bboxs for all anchors {&&} in the RPN where subscript is the anchor index. In experiments, Multi-Orsu Thresholding (Liao et al. (2001)) algorithm can efficiently separate the foreground masks with trivial overhead. More than that, models such MONet, IODINE and GENESIS tends to decompose the entire scene which make the object representation hard to learn. Intuitively, only the surrounding area is necessary for the foreground object separation. Since feature maps outside the ROI Mask is set to O, the object representation learning can focus only on the area selected by bboxs. This approach can provide a better segmentation from its surrounding area.
3.5 Loss FUNCTION
The system is training end-to-end with the following loss:
K
L(d;; 0; v/; = log
m&po p + B DKI(go(zala, mm) (z)l((()))
K
Y
DKL ( (mxx)z)))
(3)
+ 8 Σ PiiLss(Pi.PP
D:) + X Pibbrggli Ee)
Nreg Σ
The first term of the loss is the VAE decoder negative log-likelihood loss. This loss makes the reconstructed objects approximate to the scene inside the segmentation masks. The second term of the loss is the Kullback-Leible (KL) divergence between the latent variable posterior distribution (= x m&) and the latent variable prior distribution p(=x) (unit Gaussian prior). The first two terms are derived from the standard VAE's variational lower bound (ELBO) (Kingma & Welling (0013)
The third term is the KI. divergence between the distribution of network generating segmentation masks qu (mx/ar) and the VAE's decoded segmentation masks distribution pe [ This term forces the VAE to generalize the distribution of the segmentation masks {mæ} coming from the spatial attention network.
The last two terms of loss are the bounding box (bbox) regression loss from the Regional Proposal (RPN) (Ren et al. (2013). They penalize the difference between the pseudo bbox coor- dinates { which from foreground segmentation masks and the bbox coordinates for all anchors (1+} from the RPN. The penultimate term of the loss is a binary classification log loss between the predicted probability of anchor being an object (pş) and the pseudo label of anchor é being positive or negative (p:). The detail about assigning positive/negative labels to anchor is described in Faster R-CNN (Ren al. (2015)). The last term of the loss smooth L1 loss (Girshick) (2015)) between the RPN proposed bboxes and bboxes from foreground segmentation masks. The Nreg in the last term is the number of anchors in an image. The last 4 terms of the loss are weighted by the tuneable hyperparameter 8, 5 and X.
4 RESULTS
We evaluate our model two datasets:
5

---

Under review as a conference paper at ICLR 2021
1) CLEVR (Johnson et al. (2017) contains photorealistic simple rendered 3D objects. The source
images and labels come from the Multi-Object Datasets (Kabra et al. (2019)) and consist of 50000
images with resolution of 240*320. We use the method proposed in MONet (Burgess et al. (2019)
to transform the images into 128* 128. The dataset is divided into 48400 images for training and 1600 images for testing. As mentioned in (Greff al. (2019), only images with 3-6 objects (inclu-
sive) is used. In the experiment on CLEVR, the number of components K is set to 7 (6 slots for the foreground and I slot for the background) for all compared models (except SPACE).
2) Multi-dSprites (Matthey et al. (2017), Kabra et al. (2019) consists of 61600 images with a resolution of 64*64. 64. It contains 2D colourised sprites as the foreground objects and the grayscale background with uniform randomly brightness in each scene. Each scene contains 2-5 random sprites which vary in terms of shape (square, ellipse, or heart), color (uniform saturated colors), scale (continuous), position (continuous), and rotation (continuous). We divide this dataset into 60000 images for training and 1600 images for testing- the experiment on Multi-dSprites, the number of components K is set to 6 (5 slots for the foreground and slot for the background) for compared models (except SPACE).
The compared models in the evaluation are given by:
MONet, IODINE,: Our baselines in terms of ARI. These models do not generate fore- ground object position and scale representation (i.e. bbox).
This is a variant of MONet. It replaces the UNet with the spatial attention network (ResNet18 + FPN + segmentation head) in R-MONet(Lite), Since it does not contain object detection branch and proposed sule-supe loss, it is used to prove that the performance gain of R-MONet(Lite) does not come from the difference of spatial attention networks.
MONet(ResNet18 FPN):
MONet(ResNet18 FPN + UNet): This is a variant of MONet. It replaces the UNet with the spatial attention network (ResNet18 + FPN + UNet like connection) in R- MONet(UNet). Since it does not contain object detection branch and proposed self- supervised loss, it is used to prove that the performance gain of R-MONet(UNet) does not come from the difference of spatial attention networks.
SPACE: Our baselines evaluated with both ARI and mAP.
R-MONet(ROI Align): This is the R-MONet framework adapts the ROI Align and the segmentation head similar to Mask R-CNN.
R-MONet(Lite:: This is the R-MONet framework adapts the ROI Masking and the seg- mentation head similar to Mask R-CNN.
R-MoNet(UNet): This is the R-MONet framework adapts the ROI Masking and the im- proved segmentation head proposed in this paper.
3) Quantitative Evaluation Metrics:
To quantify the decomposition and segmentation quality, we measure the similarity between the ground-truth foreground segmentation masks and the foreground segmentation masks output from the spatial attention network with the Adjusted Rand Index(ARI) (Rand (971)). ARI measures the clustering similarity ranges from -1 to 1. The random labeling will have a ARI score close to O. The ARI score will be close to for perfect matching. The advantage of ARI is it can handle arbitrary permutations in the output and target clusters.
To quantify the quality of foreground object position and scale representation (i.c. bounding boxes), use the mean average precision (mAP) (Lin al. (2014) from the MS-COCO metrics (Lin al. 2014)). ). In this evaluation, all foreground objects will be treated as the same class.
4.1 RESULTS ANALYSIS AND ABLATION STUDY
MONet vs R-MONet
As shown in Table I both R-MONet(Lite) and R-MONe((UNet) surpass MONet on CLEVR dataset in terms of ARI. This is because MONet's loss does not explicitly restrict multiple objects with similar colors existing in the same mask. As we can see in Figure ( g) MONet can not separate close objects with similar colors. More than that, since MONet performs segmentation on the entire image, it often suffers from small objects. When two objects visually connected with each other,

---

Under review as a conference paper at ICLR 2021
Table 1: Segmentation performance comparison of MONet, MONec(esNNe1/++FPN) 10-
DINE, SPACE, MONecRReNNerIB4FPNNUNNG R-MONei(ROI Align), R-MONe((Lit), R- MONet(UNee).
Model
Dataset
ARI
MONer
CLEVR
0,927
MONet (ResNet18 FPN)
CLEVR
0.822
MONet (ResNet18 FPN + UNet)
CLEVR
0.825
IODINE
CLEVR
0.962
SPACE
CLEVR
0.934
R-MONet (ROI Align)
CLEVR
0,878
R-MONet (Lite)
CLEVR
0.949
R-MONet (UNet)
CLEVR
0.981
MONet
M-dSprites
0.944
MONet (ResNet18- FPN)
M-dSprites
0.607
MONet (ResNet18 FPN + UNet)
M-dSprites
0.853
IODINE
M-dSprites
0.724
SPACE
M-dSprites
0.844
R-MONet (ROI Align)
M-dSprites
0.633
R-MONet (Lite)
M-dSprites 0,801
R-MONet (UNet)
M-dSprites 0.951
Table 2: The foreground object bounding boxes comparison (bounding box mean average precision) of SPACE, R-MONei(ROI Align), R-MONet(L.te), R-MMONe(UNNe),
Model
Dataset
m.PP50:95
mAPso
mAPPs
SPACE
CLEVR
0.592
0,863
0.679
R-MONet (ROI Align)
CLEVR
0.488
0.865
0.496
R-MONet (Lite)
CLEVR
0.611
0.935
0.730
R-MONet (UNet)
CLEVR
0.645
0.969
0.775
SPACE
M-dSprites
0.309
0.757
0.174
R-MONet (ROI Align)
M-dSprites
0.319
0.711
0.237
R-MONet (Lite)
M-dSprites
0.427
0.802
0.419
R-MONet (UNet)
M-dSprites
0.435
0.839
0.407
MONet may group them together as a single object even with different colors. This case is shown in Figure () We also find in Figure 10), MONet may split single object in multiple masks under certain lighting effects such as reflection or shadow. On the contrary, both R-MONet(L and R-MONes(UNNe) will not have this problem. Since segmentation of R-MONet is only performed inside ROI, the segmentation is more accurate. With the help of Multi-Otsu thresholding method and loss from pscudo bbox, proposed self-supervision will split the ROIs which contain multiple objects.
As shown in Table I. R-MONe((UNet) still surpasses MONet while R-MONe((Lite)'s performance with CLEVR, the former
is worse than MONet on Multi-dSpritts dataset. Comparing dataset contains more sharp edges and more rectangular shapes. Since (--MOet(Lite)) only extracts conv3 layer feature maps from FPN, the segmentation head can not generate sharp edges. With help of UNet like connection, MONet and proposed improved segmentation head in R-MONet(UNNe) can produce segmentation masks with sharper edges and more complex shapes. Because Multi-dSprites is a 2D dataset, does not have complex lighting effects, MONet is slightly better on this dataset. However,
still some problems in output of MONet such multiple objects in the same mask (Figure 12 [3) or single object split (Figure II
15 16).
Besides the segmentation performance, R-MONet converges faster than MONet during training due to the defects of MONet's loss. We use the ARI evaulated on test set as measurement conver- gence. Details can be found in Figure T8 Figure T9

---

Under review as a conference paper at ICLR 2021
MONe(ResNeeI8-FFN) and MONee(ResNetLL-FPN--NNN
Through experiments on MONocResNetL8+FFN and MONecRResWNeLLS--FN--UNO we test whether the performance improvement is related to our framework comes from the back- bone difference. We change the input channel of the first convolution layer in ResNet18 from 3 to 4 and remove object detection branch. This change makes the backbone in our model be able to use as the recurrent attention module in MONet. As we can see in Table [ after switching the UNet like backbone with ( or RResNet/8=FPN+NNN mance downgrades in terms of ARI on both datasets. Because the UNet in original MONet is the same as (ONNe(ResNet18tUNNe
the perfor-
the problem may come from the FPN. Since FPN adds feature maps instead of concatenation, it tends to mix features from different layers. This is and
not helpful for segmentation task. Even with different backbones, MONe(RRsNNet1+++ MONet( ResNetl 8+ 8+FNN+UNet) still suffer from the same problems such as multiple objects in the same mask or single object split. In summary, combining object detection branch and proposed self-supervised method can effectively eliminate common problems of MONet.
IODINE As shown in Table D IODINE has higher quantitative segmentation performance than MONet on CLEVR dataset in terms of ARI. IODINE has much lower performance than MONet on Multi-dSprites dataset due to the soft edges in segmentation masks. This is expected and consistent with what is reported in its paper. From the qualitative results in Figure 2 IODINE seems to mix foreground objects with parts of background on CLEVR dataset. IODINE has better foreground separation on Multi-dSprites dataset but failed to generate accurate foreground masks. Our proposed model R-MONet(UUee) still surpasses IODINE on both datasets in terms of ARI.
SPACE As shown in Table and Table 2 our proposed model R-MONe((Lite) and R-MONe((UUNe)) have better segmentation performance in terms of ARI and object localization performance in terms of mAP. This consistent with the qualitative results. On CLEVR dataset, SPACE captures part of the object in Figure 7 9) On Multi-dSprites dataset, SPACE tends to split the single object into
objects into multiple parts (Figure 3 (2).
ROI Masking vs ROI Align Another ablation study we did is the comparison between R- MONet(ROI Align), R-MONet(Lite) and R-MOONe(UNNe). As seen in Table D and Table 2 the models using ROI Masking outperform the model using ROI Align in terms of ARI and mAP. This may be because the self-supervised bounding boxes generated by the Multi-Otsu method are not as accurate as the human labels in the supervised learning. ROI Masking is more robust than ROI Align when training with inaccurate pscudo bounding boxes because of the enlarged ROI. More than that, enlarged ROI can help the segmentation head find the object outside the proposed bounding boxes. When the bounding box only covers part of the foreground object, the enlarged foreground object mask can keep enlarging the proposed ROI until full object is covered with the pseudo bounding box label. Comparing R-MONet(Lite) and R-MOONe(UUNet), the UNet like segmentation head clearly outperforms the other segmentation head due to the skip connection and the added lower level fea- ture maps.
R-MONet(Lite) vs R-MONet(UNNt) No matter is quantitative result in terms of ART and mAP or qualitative result, proposed improved segmentation head achieves better results. Clearly, proposed improved segmentation head can generate sharper edge and preserve more low level features than the segmentation head proposed in Mask-RCNN or S4Net. In terms of m.PPs on Multi-dSprites dataset, R-MONet(Lite) outperforms R-MONet(UNet)) This may be because of the highly occluded objects in Multi-dSprites dataset. A large occluded objects may be falsely detected several small objects. In this case, a better segmentation does not guarantee a better object localization.
CONCLUSIONS
We propose R-MONet, a generative single stage framework for unsupervised scene decomposition and representation learning. By ensuring the consistency between foreground object bounding boxes and foreground segmentation masks, the model are able to decompose the foreground objects and learn better object geometric representations in the complex scene. After evaluation on CLEVR and Multi-dSprites, R-MONet achieves better quantitative decomposition performance than the of art in terms of ARI and mAP. The future direction is to adapt this model with unsupervised visual representation learning on natural image datasets.
8

---

Under review as a conference paper at ICLR 2021
image
recons
bbox
mask
mask
mask 2
mask
mask
mask
MONet
MONet (ResNet18++ FPN)
MONet (ResNet18+FFN+UUNe))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet
(UNet)
Figure 2: Qualitative comparison of scene reconstruction, bbox and foreground segmentation masks of MONet, MONet(ResNer18 + FPN), NeNee(ResNet18 8 FPN UNet), IODINE, SPACE, R- MONei(ROI Align), R-MONet(Lite), R-MONe((UNet) on CLEVR dataset. More results can be seen in Appendix A
image
recons
bbox
mask
mesk
mask 2
mask
mesk
MONet
t
MONet
MosNNe18+FFPM
MONet
(ReNNe11++FPN-UNNe))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 3: Qualitative comparison of scene reconstruction, bbox and foreground segmentation masks of MONet, M + FPN), MoNet(ResNet18 FPN UNet), IODINE, SPACE, R- MONeu(ROI Align), RtMe)et(LLite), R-MoNN(((UNe)) on Multi-dSprites dataset. More results can be seen in Appendix A

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv:1901.11390 2019.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 3412-3420, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Confer- ence on Learning Representations, 2020. URL
BkxfaTVFwH
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hin- ton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Ruochen Fan, Ming-Ming Cheng, Qibin Hou, Tai-Jiang Mu, Jingdong Wang, and Shi-Min Hu. S4net: Single stage salient-instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6103-6112, 2019.
Ross Girshick. Fast r-cnn. In Proceedings the IEEE international conference computer vision, pp. 1440-1448, 2015.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017. 2017
Klaus Greff, Raphaël Lopez. Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint urXiv:1903 004400050 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
ZQ John Lu. The elements of statistical learning: data mining, inference, and prediction. Journal of the Royal Statistical Society: Series A (Statistics in Society), 173(3):693 694, 2010.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-29161 2017.
Rishabh Kabra, Chris Burgess, Loie Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https//gghhbb.ommdeppnn////iii object-datasets/ 2019.
Diederik Kingma and Max Welling. arXiv: 2013.
Auto-encoding variational bayes.
arXiv preprint
Ping-Sung Liao, Tse-Sheng Chen, Pau-Choo Chung, et al. fast algorithm for multilevel thresh- olding. J. Inf. Sci. Eng., 17(5):713-727, 2001.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
10

---

Under review as a conference paper at ICLR 2021
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
URL httpp:://ppereeeeeee wttpriewwwee/eeeeeee Bewp:/nee//oormmee Fow.:/mmndddd-
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Pro- ceedings of the 18th ACM international conference on Multimedia, pp. 1485-1488, 2010.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. URL https://githhb.. hmm/leepminddppprresedaad/////ee 05-08), 2017.
2018-
Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th International Conference on Puttern Recognition (ICPR volume pp. 850-855. IEEE, 2006.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the Ameri- can Statistical association, 666336):846-850, 1971.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Jayaram Sethuraman. A constructive definition of dirichlet priors. Statistica sinica, pp. 639-650, 1994.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10335 10353, 2018.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint 1oX10:1901.07017, 2019.

---

Under review as a conference paper at ICLR 2021
A
ADDITIONAL PLOTS
image
MONet
MONet [ResNet18+
MONet
(
IODINE
SPACE
R-MONet (ROI Align)
R-MONet
(Lite)
R-MONet (UNet)
Figure 4: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
image
recons
bbox
mask
mask
mask 2
mask
mask 4
mask
MONet
MONet
[ResNet18+FPN)
MONet
(ReSNet18+FPN+UNett
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 5: Additional qualitative comparison scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
12

---

Under review as a conference paper at ICLR 2021
image
recons
bbox
mask
mask
mask 2
mask
mask 4
mask
MONet
MONet (ResNet18+FPNN
MONet
(ResNet18+FNN+UNG
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 6: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
image
recons
bbox
mask
mask
mask 2
mask
mask 4
mask
MONet
MONet
BeesNet18+FPM
MONet
(ReNNet18+FNN+UNNN)
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 7: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
13

---

Under review as a conference paper at ICLR 2021
image
recons
bbox
mask
mask
mask 2
mask
mask 4
mask
MONet
MONet [ResNet18+FPNN
MONet
(ResNet18+FNN+UNG))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 8: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
image
recons
bbox
mask
mask
mask 2
mask
mask 4
mask
MONet
MONet
PosNNet18+FPM
MONet
(ReSNet18+FNN+UNN))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 9: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
14

---

Under review as a conference paper at ICLR 2021
imagee
recons
bbox
mask
mask
mask
mesk
mask
mesk
MONet
MONet (ResNet18+
MONet
(ReSNet11+FFNN+UNNet)
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 10: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on CLEVR dataset.
image
recons
bbox
mask
mask
mask 2
mask
mask
MONet
MONet
(ResNet18+FPN)
MONet
(ReSNet1B+FPN++NNet)
IODINE
SPACE
R-MONet
(ROI Align)
R-MONet
(Lite)
R-MONet (UNet)
Figure 11: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on Multi-dSprites dataset.
15

---

Under review as a conference paper at ICLR 2021
Image
bbox
MONet
MONet (N))N
MONet (ResNet18+FPN+UNNE)
IODINE
SPACE
R-MONet (ROI Align)
R-MONet
(Lite)
R-MONet (UNet)
Figure 12: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on Multi-dSprites dataset.
bbox
mask
mask
MONet
MONet
(ResNet18+FPa)
MONet
(ReNNet18+FNN-UNet)
IODINE
SPACE
R-MONet (ROI Align)
R-MONet
(Lite)
R-MONet (UNet)
Figure 13:
Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on Multi-dSprites dataset.
16

---

Under review as a conference paper at ICLR 2021
bbox
MONet
MONet FN))
MONet (ResNet18+FPN+UNN))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 14: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on Multi-dSprites dataset.
bbox
mask
mask
MONet
MONet
[ResNet18+FPM)
MONet
(ResNet18+FPN+UNNe))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet (Lite)
R-MONet (UNet)
Figure 15: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on Multi-dSprites dataset.
17

---

Under review as a conference paper at ICLR 2021
image
MONet
MONet (ResNet18+FPN)
MONet [ResNet18+FPN+UNNC))
IODINE
SPACE
R-MONet (ROI Align)
R-MONet
(Lite)
R-MONet (UNet)
Figure 16: Additional qualitative comparison of scene reconstruction, bbox and foreground segmen- tation masks on Multi- Aulti-dSprites dataset.
masked recon II
masked recon
masked recon
masked recon
masked recon
masked recon 3
bbox
masi D
mask:
mask
mask
mask 4
mask: 5
epoch480
sample1
epoch480 sample2
epoch864 sample1
epoch864
sample2
epoch7008
sample1
epach7008
sample2
epoch19296
sample1
epoch19296
sample2
Figure 17: Qualitative comparison of R-MoNes(UNNet's masked object reconstruction, bbox and foreground segmentation masks at epoch 480, epoch 864, epoch 7008, epoch 19296 on CLEVR dataset. the initial stage like epoch 480, ROIs are random across the image. The spatial attention network tends to learn segmentation inside the ROIs. After spatial attention network learns the rough segmentation masks (epoch 864), the pseudo ground truth bboxs generated from rough segmentation mask guide object detection branch to find more accurate ROIs. In this stage, if segmentation masks contain more than one object, pseudo ground truth bbox will separate them with Multi-Otsu algorithm. At the middle stage (epoch 7008), the evolving segmentation masks help VAE to learn object appearance representations. In the last stage (epoch 19296), segmentation masks, bboxs and object appearance representations from VAE keep evolving the same time.
18

---

Under review as a conference paper at ICLR 2021
Img
mask
masic
sample1
sample2
sample3
S
sample4
R
M
sample5
sample6
sample7
sample8
Figure 18: Qualitative comparison of MONet's masked object reconstruction, bbox and foreground segmentation masks at epoch 47040 on CLEVR dataset. During training, since the loss of MONet does not prevent multiple objects showing up in the same mask, MONet tends distribute the objects with the same reconstructed color into the same mask. This problem may slow down MONet training.
Figure 19: Quantitative training convergence speed comparison between MONet, R-MONet(Lite) and UNMt). We compared the ARI performance growth speed in terms of batches and training time. On CLEVR dataset and Multi-dSprites dataset, R-MONet converges much faster than MONet. This may be because MONet uses the entire scene as the ROI and it is harder for attention network to obtain a proper segmentation for each object compared with regional ROI R-MONet. Due to the better segmentation quality, R-MONet((UNe) converges even faster than R- R=MONe((Lite)
19

---

Under review as a conference paper at ICLR 2021
masked recon
masked recon
masked recon 2
masked recon 3
masked recon 4
masked recon
Fmg
mask
mask
mask
mask
mask
mask 5
sample1
sample2
sample3
sample4
sample5
sample6
sample7
sample8
Figure 20: Qualitative comparison of R-MONe((UNet'ss (without the self-supervised loss and ob- ject detection branch) masked object reconstruction, bbox and foreground segmentation masks on CLEVR dataset. This model performs segmentation on the entire image and generates object masks parallel. The ARI is nearly zero since all object segmentations are one mask. The spatial atten- tion module is good at segmenting objects from the background but unable to separate objects from each other. This proves that the loss MONet does not prevent multiple objects from showing up in the same mask and the effectiveness of proposed self-supervised loss.
masked
masked
sample1
sample2
sample3
sample4
sample5
sample6
sample7
sample8
sampleg
Figure 21: Qualitative comparison of R-MONet(UNee) masked object reconstruction, bbox and
foreground segmentation masks MS-COCO 2017 dataset (Lin al. (2014)
We use the pretrained ResNet18+FPN on ImageNet (Deng et al. (2009) provided in torchvision
MMrrell & Rodriguez (2010). Unfortunately, the proposed model can not achieve the same perfor- mance as the supervised models. The VAE used can not generate complex object in the scene. It focuses more on the region which has high contrast with its surrounding areas.
20

---

Under review as a conference paper at ICLR 2021
B IMPLEMENTATION DETAILS
Stick-Breaking Process. {<x } is the segmentation intermediate outputs. {m&} } is the segmentation masks (K-1 foreground and I background). 8y represents the scope which is the proportion of pixels still unexplained.
80 = 1
(4)
8k - 8k-1 ank) k E [1, 1]
(5)
m Sk 1kk [1, K 1]
(6)
mk 8K-
VAE Decoder Negative Log Likelihood.
k
log (a( {zx}) = - log (aPo(2/2N
K-1
log N (xx obg)- - log Σ
MNN 030)
k=
(8)
(x TK 2
=lgog mk
expl
bb 2
20 bg
2
K-1
(x: Tx) )2
log
m
exp(
0ig VT
2 19
k=1
21

---

Under review as a conference paper at ICLR 2021
ResNett8
Lapar
Segneridin
Improved Sogmentation
Masking
Unet Like Sep Connection
(ax)
Original ROI Masking
(ax)
Align
Figure 22:
The feature map flow of the improved segmentation head. (a) The input image is passed into ResNet18 and (1-v layers extracted for the later The c2nv(2-5) layers are passed into FPN for the bounding boxes generation in RPN. After the FPN, the conv5 layer is passed into MLP compress its feature space. During the region feature selection, only conv [- 3} layers are transformed with ROI Masking method. The output of different layers are combined together like UNet to preserve low level features. (b) The original ROI Masking method enlarges the selected region by applying the ternary masks to the feature maps and it is only applied to the features from the conv3 layer. The transformed feature maps keep the conv3's dimensions. (c) The ROI Align method only extracts the feature maps in the selected area and transforms them into the fixed dimensions. The segmentation head similar the one used the Mask R-CNN when adapting ROI Masking and ROI Align.
22

---

Under review as a conference paper at ICLR 2021
ResNet18 FPN + RPN. The implementations of ResNet18, Feature Pyramid Network (FPN), Region Proposal Network (RPN) and bounding box loss used in RPN are adapted from torchvision (0.4.1) with some detail adjustment.
MONet / Spatial Broadcast VAE. The implementations of MONet and Spatial Broadcast VAE use some ideas from foomgiihubbccm com baudm baudm/MONet MMNNeM
Multi-Otsu Thresholding, Multi-Ore Austi Thresholding method a thresholding algorithm that is used to separate the pixels of an input image into several different classes by maximizing the between- class variance pixel intensity (grey level). The implementations of Multi-Otsu Thresholding method and the method for generate bounding boxes with multiple thresholds
from scikit-image (0.16.2). Before applying Multi-Otsu method, the values under 0.001 is set to 0. The score for the pseudo ground truth bounding boxe & the mean value of mn in the bounding box area.
IODINE. The implementation of IODINE is from https::/github..cmmm//t wwwpm//hhxxxanan lin/
IODINE
SPACE. The implementation of SPACE is from sttps:///ithub.com wttp://c///hiiaanniii.
SPACE
Hyperparameters on CLEVR and Multi-dSprites
For presented models, we performed hyperparameter search and choose the results for the best settings. IODINE, SPACE, MONet, R-MONet(Lite) and R-MONet(UNee) are sensitive to VAE decoder scale (standard deviation). Other hyperparameters are robust in a reasonable range.
23

---

Under review as a conference paper at ICLR 2021
R-MONet(Lite)
Module
Name
Value
CLEVR
input image dimension
(128, 128)
image normalization
batch size
50
optimizer
Adam
momentum
learning rate learning rate schedular learning rate decay interval learning rate reset decay warmup
0.0001
Cosine
200 epochs
0.05
10 epochs
ROI Masking
expand ratio
0.3
RPN
anchor size
(16, 32, 64, 128, 256)
RPN
anchor aspect ratio
(0.5, 1.0, 2.0)
RPN
NMS threshold inside each layer 0.3
RPN
NMS threshold for output bbox
0.3
RPN
NMS top
6
RPN
pre layer NMS top post layer NMS top n foreground IOU threshold background IOU threshold
100
RPN
20
RPN
0.7
RPN
0.3
RPN
batch size per image positive fraction prediction score threshold bbox minimum size bbox maximum h/w ratio bbox maximum w/h ratio
256
RPN
0.5
RPN
0.5
Multi-Otsu
5
Multi-Otsu
3
Multi-Ots
3
Multi-Otsu
NMS threshold
0.7
Multi-Otsu
number of bins
5
VAE
z dimension
16
VAE
background scale o
0.06
VAE
foreground scale o
0.10
VAE
component number K
7
3
0.5
0.5
1
A
24

---

Under review as a conference paper at ICLR 2021
R-MONee(UNet)
Module
Name
Value
CLEVR
input image dimension
(128, 128)
image normalization
batch size
50
optimizer
Adam
momentum
learning rate learning rate schedular learning rate decay interval learning rate reset decay warmup
0.0001
Cosine
50 epochs
0.1
5 epochs
ROI Masking
expand ratio
0.3
RPN
anchor size
(16, 32, 64, 128, 256)
RPN
anchor aspect ratio
(0.5, 1.0, 2.0)
RPN
NMS threshold inside each layer 0.3
RPN
NMS threshold for output bbox
0.3
RPN
NMS top
6
RPN
pre layer NMS top
100
RPN
post layer NMS top foreground IOU threshold
20
RPN
0.7
RPN
background IOU threshold
0.3
RPN
batch size per image positive fraction prediction score threshold bbox minimum size bbox maximum h/w ratio bbox maximum w/h ratio
256
RPN
0.5
RPN
0.5
Multi-Otsu
5
Multi-Otsu
Multi-Otsu
3
Multi-Otsu
NMS threshold
0.7
Multi-Otsu
number of bins
5
VAE
dimension background scale o foreground scale d
16
VAE
0.06
VAE
0.10
VAE
component number K
0.5
7
0.5
1
25

---

Under review as a conference paper at ICLR 2021
MONet
Module Name
Value
CLEVR input image dimension
(128, 128)
image normalization
batch size
optimizer
Adam
momentum
learning rate learning rate schedular learning rate decay interval
0.0001
Cosine
200 epochs
learning rate reset decay
0.05
warmup
10 epochs
VAE
dimension
16
VAE
background scale a
0.06
VAE
foreground scale d%
0.10
VAE
component number
B
0.5
0.5
IODINE
Module
Name
Value
CLEVR
input image dimension
(128, 128)
image normalization batch size optimizer
20
Adam
learning rate component number K
0.0003
iteration T
latent dimension
64
Refine Network convolution output channel
64
Refine Network convolution layers
Refine Network mlp units
256
Refine Network kernal size
Refine Network stride
Decoder
convolution output channel
64
Decoder decoder standard deviation
0.1
Decoder convolution layers
Decoder
kernal size
26

---

Under review as a conference paper at ICLR 2021
SPACE
Module Name
Value
CLEVR input image dimension
(128, 128)
image normalization
batch size
fg optimizer
RMSprrp
bg optimizer
Adam
fg learning rate
0.00001
bg learning rate
0.001
grid
8
clip norm
fg VAE
fg standard deviation
0.11
fg VAE
z_pres start step
4000
fg VAE
z_pres end step
10000
fg VAE
z-pres start value
0.1
fg VAE
z-pres end value
0.01
fg VAE
scale mean start step
0
fg VAE
z scale mean end step
20000
fg VAE
7_scale mean start value
-1.0
fg VAE
z_scale mean end value
-2.0
fg VAE
z_aclee std value
0.1
g VAE
tau start step
0
gg VAE tau end step
20000
g VAE tau start value
2.5
fg VAE tau end value
0.5
bg VAE background component number K
bg VAE bg standard deviation
0.09
boundary loss
true
boundary loss end step
50000
27

---

Under review as a conference paper at ICLR 2021
R-MONet(Lite)
Module
Name
Value
M-dSprites
input image dimension
(64, 64)
image normalization
(0.5, 0.5)
batch size
64
optimizer
Adam
momentum
learning rate learning rate schedular learning rate decay interval learning rate reset decay warmup
0.0001
Cosine
200 epochs
0.05
10 epochs
ROI Masking
expand ratio
0.3
RPN
anchor size
(16, 32, 64, 128, 256)
RPN
anchor aspect ratio
(0.5, 1.0, 2.0)
RPN
NMS threshold inside each layer 0.3
RPN
NMS threshold for output bbox
0.3
RPN
NMS top
5
RPN
pre layer NMS top post layer NMS top n foreground IOU threshold background IOU threshold
100
RPN
20
RPN
0.7
RPN
0.3
RPN
batch size per image positive fraction prediction score threshold bbox minimum size bbox maximum h/w ratio bbox maximum w/h ratio
256
RPN
0.5
RPN
0.5
Multi-Otsu
3
Multi-Otsu
3
Multi-Ots
3
Multi-Otsu
NMS threshold
0.7
Multi-Otsu
number of bins
5
VAE
z dimension
16
VAE
background scale o
0.09
VAE
foreground scale o
0.11
VAE
component number K
6
3
0.5
0.5
1
A
28

---

Under review as a conference paper at ICLR 2021
R-MeNee(UNee)
Module
Name
Value
M-dSprites
input image dimension
(64, 64)
image normalization batch size optimizer
64
Adam
momentum
learning rate learning rate schedular learning rate decay interval learning rate reset decay warmup
0.0001
Cosine
50 epochs
0.1
5 epochs
ROI Masking
expand ratio
0.3
RPN
anchor size
(16, 32, 64, 128, 256)
RPN
anchor aspect ratio
(0.5, 1.0, 2.0)
RPN
NMS threshold inside each layer 0.3
RPN
NMS threshold for output bbox
0.3
RPN
NMS top
RPN
pre layer NMS top
100
RPN
post layer NMS top foreground IOU threshold
20
RPN
0.7
RPN
background IOU threshold
0.3
RPN
batch size per image positive fraction prediction score threshold bbox minimum size bbox maximum h/w ratio
256
RPN
0.5
RPN
0.5
Multi-Otsu
3
Multi-Otsu
Multi-Otsu
bbox maximum w/h ratio
3
Multi-Otsu
NMS threshold
0.7
Multi-Otsu
number of bins
5
VAE
dimension
16
VAE
background scale o foreground scale d
0.09
VAE
0.11
VAE
component number K
6
0.5
7
0.5
1
29

---

Under review as a conference paper at ICLR 2021
MONet
Module
Name
Value
M-dSprites
input image dimension
(64, 64)
image normalization
batch size
64
optimizer
Adam
momentum
learning rate learning rate schedular learning rate decay interval
0.0001
Cosine
200 epochs
learning rate reset decay
0.05
warmup
10 epochs
VAE
dimension
16
VAE
background scale d
0.09
VAE
foreground scale d
0.11
VAE
component number K
0.5
0.5
IODINE
Module
Name
Value
M-dSprites
input image dimension
(64, 64)
image normalization batch size optimizer
64
Adam
learning rate component number iteration T
0.0001
6
5
latent dimension
16
Refine Network convolution output channel
32
Refine Network convolution layers
3
Refine Network mlp units
128
Refine Network kernal size
3
Refine Network stride
2
Decoder convolution output channel
32
Decoder decoder standard deviation
0.1
Decoder convolution layers
5
Decoder kernal size
3
30

---

Under review as a conference paper at ICLR 2021
SPACE
Module
Name
Value
M-dSprites
imput image dimension
(128, 128) (upscale from (64,64))
image normalization
batch size fg optimizer bg optimizer
48
RMSprop
Adam
fg learning rate
0.00001
bg learning rate grid
0.001
8
clip norm
1
fg VAE
fg standard deviation
0.11
VAB
z.pres start step
4000
g VAE
z_pres end step
10000
VAE
z-pres start value
0.1
fg VAE
z-pres end value
0.01
VAE
Z scale mean start step
0
fg VAE
7. scale mean end step
20000
VAE
7_scale mean start value
-1.0
g VAE
z.scale mean end value z.scale std value tau start step tau end step tau start value tau end value background component number K 3
-2.0
VAE
0.1
gg VAE
0
fg VAE
20000
ig VAE
2.5
fg VAE
0.5
VAE
bg VAE
bg standard deviation boundary loss
0.09
true
boundary loss end step boundary loss width
(keep boundary loss to the end of training)
2
31


Reviews:
[{"review_id": "bCfdJK8sC2a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n\nThe paper presents a method to decompose scenes into its constituent objects. This is done with a generative framework that generates both bounding boxes and segmentation masks for each object. It relies on several previously existing technologies. Its main contribution is enforcing consistency between bounding boxes and segmentation masks.\n\nPROS\n\n* Outperforms the baselines.\n\nCONS\n\n* The paper can be hard to read.\n* Contributions seem minor.\n* Good results, but on two toy datasets only.\n\nCOMMENTS\n\nThe writing should be improved, as the paper can be hard to follow. One one hand, this includes broken sentences (\"Inspired by the observation that foreground segmentation masks and bounding boxes both contain object geometric information and should be consistent with each other.\"), grammatical errors (\"It proves that there are still many useful information can be discovered in those unlabeled data.\"), and sentences which are just hard to parse (\"In the former type of models, the scene is encoded into the object-oriented disentangled spatial and appearance encoding explicitly.\"). On the other, the authors cite many concepts without introducing them in the paper (stick breaking, spatial broadcast decoder, multi-otsu thresholding, etc) which makes it non self-contained.\n\nThe paper presents what seem like engineering improvements over previous works (e.g. combining bounding boxes and segmentation masks) by adding more components to the framework, which is quite convoluted (see Fig. 1: ResNet, FPN, RPN, segmentation, VAEs, etc). It is hard to know where performance comes from, despite the ablation tests.\n\nThe experiments are limited to two toy datasets with a fixed number of simple objects (which must be known beforehand), which show no background interference and little occlusion.\n\nIn all, I do not think it meets the ICLR bar.\n\nI am not an expert on the topic so I may have missed relevant datasets/baselines.\n\nDetail: \"Region of Interest\" introduced after ROI has been mentioned several times.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OVkRfWm_SpV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a region-based approach for unsupervised scene decomposition. It extends the previous MONet by introducing the region-based self-supervised training. Instead of purely generating foreground masks in an RNN, they simultaneously predict the bounding boxes and segmentation masks using a Faster-RCNN-based framework. The supervision comes from the object reconstruction loss and the self-supervised loss of classification and regression of anchors in the RPN module. The experiments and comparisons are only conducted on the synthetic CLEVR and Multi-dSprites dataset. \n\n[Paper strength]\n- The paper is well motivated, and the proposed approach seems to be reasonable.\n- The self-supervised idea is interesting that uses the segmentation mask to get the pseudo bounding box label for object detection, which could ensure the consistency of object mask and bounding box. \n\n[Paper Weakness]\n1. The self-supervision between segmentation masks and detection bounding boxes is the main contribution. While incorporating the self-supervision into MONet is meaningful and interesting, the overall novelty does not look significant.\n\n2. Clarification of Methods:\n- How to learn $m_k$ in a self-supervise way is unclear? MONet uses spatial attention to identify the most salient object one by one, which makes senses. But here you segment all the objects in one step. How could this be possible in an unsupervised way? From the example in Fig. 2, it looks R-MONet could pick out some small and far-away objects first, which is not intuitive.\n- In Faster-RCNN, the positive/negative samples are selected by calculating the IoU threshold between the sampled bbox and the ground truth bbox. However, in this self-supervised approach, there is no ground truth bbox. Although the authors proposed to use the pseudo bbox from the segmentation mask $m_k$, how could this be reliable since $m_k$ is likely of poor quality, especially at the initial stage.\n- The selected K value is unclear. In the original MONet, the spatial attention network is an RNN-like structure, they decompose the scene step-by-step. Therefore, they define the K steps. However, in this Faster-RCNN-based framework, the objects are selected in one step, how to select the K-1 objects in all proposals?  \n\n3. Results:\n- Most of the results are with toy images. There is no result on real images.\n- There is no result to really demonstrate the effectiveness of the self-supervised loss. The author should compare their R-MONet(UNet) with the baseline of R-MONet(UNet) w/o the self-supervised loss, i.e. removing the object detection branch. Another missing baseline is MONet(UNet).\n- In Table 1, the MONet (ResNet18+FPN) is 10 percent lower than the original MONet. Does this means the network structure has a greater influence on the performance than the self-supervision component.\n- In Table 1, the R-MONet(Lite) performs worse. Once again, I guess this poor performance comes from the network structure, as the input image is 64*64, the Resnet downsamples the image to a very low resolution which losses the spatial information.\n- The visual results cannot demonstrate the advantage of the proposed approach. For example, in Figure 3, the visual performance of MONet and R-MONet(UNet) are quite similar.\n\n---\n---\nUpdate:\nIn general, I am happy with the authors' responses. They did show the advantage of the introduced self-supervised loss. Although the self-supervised loss is intuitive, incorporating it into MONet is non-trivial and it does outperform MONet. Despite that such self-supervised works are hard to work on real scenes, this paper does have some merits. I am willing to increase my rating.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "C0TtIhJRI_W", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a variation of the MONet model where an additional Region Proposal Network generates bounding boxes for various objects in the scene. An additional loss is introduced during training to make the segmentations produced by the MONet segmenter consistent with the proposed bounding boxes. Results are demonstrated on multi d-Sprites and CLEVR with modest performance gains.\n\nThe paper is somewhat middle of the road in most aspects - the proposed method is, in my opinion, only a slight variation on the existing MONet model. Though presented clearly, I don't feel that adding that loss makes the model better in any fundamental way, even if performance numbers are slightly better in some circumstances. Furthermore, though there is some ablation analysis, I feel the level of analysis of the results is sub-par - when a relatively simple variation of a model like here is proposed I would want to see an effort to analyse the contribution beyond how it affects the numbers - do we learn anything new by introducing the variation? does it tell us some fallacy or failure of the original model and if so, does it fix it? These are lacking here.\n\nA few more concrete points:\n\n* In Table 1 - why is the ResNet18 + FPN missing from d-Sprites dataset? This ablation is probably the single most important experiment present in the paper - I would want to see it reported on both datasets.\n* In general - I feel the performance on these datasets is quite saturated and I hope to see results on more challenging data in the future - the proposed method included\n* There is very little discussion about the choice of hyper-parameters in the paper - how were they chosen? is the system sensitive to these choices?\n\nPost rebuttal comments:\n\nThank you authors for the detailed response - I think some of of my concerns have been answered - the paper may be a valid contribution to the community and I am raising my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has good contributions to a challenging problem, leveraging a Faster-RCNN framework with a novel self-supervised learning loss. However reviewer 4 and other chairs (in calibration) considered that the paper does not meet the bar for acceptance. The other reviewers did not champion the paper either, hence i am proposing rejection. 

Pros:
- R1 and R3 agree that the proposed model improves over related models such as MONET.
- The value of the proposed self-supervised loss connecting bounding boxes and segmentations is well validated in experiments.

Cons:
- R4 gives good suggestions that may be useful to reach a broader readership, namely introducing more of the concepts used in the paper., e.g. "stick breaking, spatial broadcast decoder, multi-otsu thresholding" so it becomes more self-contained. R4 also suggests improving the writing more generally.
- R4 still finds the proposed "method quite complex yet derivative" after the rebuttal.
- All reviewers complain about lack of experiments in real data, but the authors did revise their paper and add some coco results in the appendix. These could be part of the main paper in a future version.
Decision: Reject