unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning
Authors: Yafei YANG
Zihui Zhang
Bo Yang
Venue: ICML 2025 poster (2025)
Keywords: unsupervised object segmentation
object-centric representation
Primary area: deep_learning->other_representation_learning
TL;DR: We propose a two stage pipeline for unsupervised multi-object segmentation on single images by learning and reasoning with three level object-centric representations.

Abstract: We study the challenging problem of unsupervised multi-object segmentation on single images. Existing methods, which rely on image reconstruction objectives to learn objectness or leverage pretrained image features to group similar pixels, often succeed only in segmenting simple synthetic objects or discovering a limited number of real-world objects. In this paper, we introduce unMORE, a novel two-stage pipeline designed to identify many complex objects in real-world images. The key to our approach involves explicitly learning three levels of carefully defined object-centric representations in the first stage. Subsequently, our multi-object reasoning module utilizes these learned object priors to discover multiple objects in the second stage. Notably, this reasoning module is entirely network-free and does not require human labels. Extensive experiments demonstrate that unMORE significantly outperforms all existing unsupervised methods across 6 real-world benchmark datasets, including the challenging COCO dataset, achieving state-of-the-art object segmentation results. Remarkably, our method excels in crowded images where all baselines collapse. Our code and data are available at https://github.com/vLAR-group/unMORE.

unMORe: Unsupervised Multi-Object Segmentation via Center- conter-Bounnary Reasoning
Yafei Yang Zihui Zhang Bo Yang" 12
Abstract
teaching on site. Inspired by this efficient skill of perceiving objects and scenes, we aim to introduce new framework to identify multiple objects from single images by learning object-centric representations, rather than relying on costly scene-level human annotations for supervision.
We study the challenging problem of unsuper- vised multi-object segmentation on single images. Existing methods, which rely on image recon- struction objectives to learn objectness or lever- age pretrained image features group similar pixels, often succeed only in segmenting simple synthetic objects discovering limited number of real-world objects. In this paper, we introduce unMORE, a novel two-stage pipeline designed to identify many complex objects real-world images. The key to our approach involves ex- plicitly learning three levels of carefully defined object-centric representations in the first stage. Subsequently, our multi-object reasoning module utilizes these learned object priors to discover mul- tiple objects in the second stage. Notably, this rea- soning module is entirely network-free and does not require human labels. Extensive experiments demonstrate that unMORE significantly outper- forms all existing unsupervised methods across 6 real-world benchmark datasets, including the challenging COCO dataset, achieving state-of-the- art object segmentation results. Remarkably, our method excels in crowded images where all base- lines collapse. Our code and data are available at http://ggttubbcoooLLARRRR-R-
Existing works for unsu- pervised multi-ohject seg- mentation mainly consist of two categories: 1) Slot-based methods rep- resented by SlotAtt (Lo- catello et al., 2020) and its variants (Sajjadi et al., 2022; Didolkar
Diffeeet Exssiase
Deftrres
Dilfazent
Figure
Object images.
al.,
2024). They usually rely on an image reconstruction ob- jective to drive the slot-structured bottlenecks to learn ob- ject representations, While achieving successful results on synthetic datasets (Karazija et al., 2021; Greff et al., 2022), they often fail to scale to complex real-world im- ages (Yang & Yang, 2022: 2024). 2) Self-supervised fea- ture distillation based methods such TokenCut (Wang et al., 2022b), DINOSAUR (Seitzer al., 2023), CutLER (Wang et al., 2023a), and CuVLER (Arica et al., 2024). Thanks to the strong object localization hints emerging from self-supervised pretrained features such as DINO/v2 (Caron et al., 2021; Oquab et al., 2023), these methods ex- plore this property to discover multiple objects via feature reconstruction or pseudo mask creation for supervision. De- spite obtaining very promising segmentation results real- world datasets such as COCO (Lin et al., 2014), they still fail to discover a satisfactory number of objects. Primarily, this is because the simple feature reconstruction or pseudo mask creation for supervision tends to distill or define rather weak objectness followed by ineffective object search, re- sulting in only few objects correctly discovered. In fact, unsupervised multi-object segmentation of single image is hard and not straightforward, as it involves two critical issues: 1) the definition of what objects are (i.e., objectness) is unclear, 2) there a lack of an effective way to discover those objects in unseen scenes.
1. Introduction
By age two, humans can learn around 300 object categories and recognize multiple objects unseen scenarios (Frank al., 2016). For example, after reading book about the An- imal Kingdom where each page illustrates single creature, children can effortlessly recognize multiple similar animals at glance when visiting a zoo, without needing additional
Shenzhen Research Institute, The Hong Kong Polytechnic University; *vLAR Group, The Hong Kong Polytcchnic University. Correspondence to: Bo Yang yang @ polyu.edu.hk>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, to tackle these issues, we propose two-stage pipeline consisting of object-centric representation learn

---

unMORE; Unsupervised Multi-Object Segmentation via Center- Center-Boundary Reasoning
Objectness Network
Multi-Ohject Reasoning
Object Existence
Objectness Network
Object Center Field
Objectness Network
Object Boundary
Distance Field
-1.0
Existence Score -1
Existence Score
Object Image
Object Center Field
Object Image
Object Boundary
(a) Object Existence
(b) Object Center Field
(c) Object Boundary Distance Field
Figure 2: The upper blocks illustrate framework. The lower blocks show three levels of object-centric representations.
ing stage followed by an effective multi-object reasoning stage, akin to a human's innate skill of perceiving objects and scenes, As illustrated in the upper left block Figure 2, in the first stage, we aim to train an objectness nefwork to learn our explicitly defined object-centric representations from monolithic object images such those in ImageNet. In the second stage, as illustrated in the right block of Figure 2, we introduce a multi-object reasoning module to auto- matically discover individual objects single images just by querying our pretrained and frozen objectness network, instead of requiring human annotations for supervision.
boundary- reasoning algorithm to iteratively regress accurate multi-object bounding boxes and masks according to the learned three levels of object-centric representations from our pretrained objectness network. Notably, the multi- object reasoning completely network-free and requires no human labels for supervision.
Our framework, named unMORE, learns object-centric representations through the objectness network, enabling unsupervised multi- reasoning on single images. Our contributions are:
We introduce a new pipeline comprising object-centric learning and multi-object reasoning, and propose three levels of explicit object-centric representations, including object existence, object center field, and object boundary distance field learned by an objectness network.
Regarding the objectness network, our key insight is that, given input image or patch, it should be able to answer three essential questions: 1) is there an object inside (i.e., object existence)? 2) if so, where is it (i.e., object loca- tion/center)? and 3) what is the object shape (i.e., object boundary)? Essentially, training such an objectness network would be analogous to the learning process of infants form- ing concepts of objects in their minds. As illustrated in that there is salient object in image
We design center- couterrb aware reasoning algorithm to iteratively discover multiple objects in single images. The algorithm is network-fres and human-labelifree.
Figure 1, #1, but images #2/#3 contain similar dogs at different loca- tions, whereas image #4 has another object with different shape boundaries. By training on such images, our object- ness network aims to explicitly capture these top-down (ex- istence/location) and bottom-up (boundary) object-centric representations. To achieve this goal, introduce three corresponding levels of objectness to learn in parallel: binary object existence score, 2) object center field, and 3) an object boundary distance field, as shown in Figure 2.
We demonstrate superior object segmentation results and clearly surpass state-of-the-ee unsupervised methods on 6 benchmark datasets including the challenging COCO.
2. Related Work
Object-centric Learning without Pretrained Features: Object-centric learning involves the unsupervised discov- ery of multiple objects in a scene. A plethora of methods have been proposed in the past years (Yuan et al., 2023). They primarily rely on an image reconstruction objective to learn objectness from scratch without needing any hu- man labels or pretrained image features. Early models aim to learn object factors such size, position, and appear- ance from images by training (variational) autoencoders (AE/VAE) (Kingma & Welling, 2014), including AIR (Es- lami et al., 2016). SPACE (Lin et al., 2020), and others (Gr- eff et al., 2016; 6; 2017: Crawford & Pineau, 2019; Burgess et al., 2019; Greff et al., 2019). Recently, with the success of slot-based methods (Locatello et al., 2020; Engelcke et al.,
With respect to the multi-object reasoning module, we aim to discover as many individual objects as possible in scene- level images. Our insight is that, given multi-object image, if a cropped patch has single valid object inside, its three levels of objectness representations must satisfy certain threshold when querying against our pretrained objectness network. Otherwise, that patch should be discarded or its position and size should be effectively updated until a valid object is included inside. this end, we introduce center-
2

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning
2020), most succeeding works (Engeleke al., 2021; Saj- jadi et al., 2022; Löwe al., 2022; Biza et al., 2023; Löwe et al., 2023; Foo et al., 2023; Brady et al., 2023; Jía et al.,
(Gall & Lempitsky, 2009; Gall et al., 2011; Qi et al., 2019; Ahn et al., 2019) learn to transform pixels/points to object centroids for better segmentation, and the works (Thanh Nguyen, 2014; Ma et al., 2010) use object bound- aries as the template for shape matching. However, these works are primarily designed for fully supervised tasks, whereas we focus on learning object-centric representations for unsupervised multi-object segmentation. In particular, our carefully designed three-level object-centric representa- tions aim to jointly describe objects in a nuanced manner, and our unique multi-object reasoning module can make full use of the learned object-centric representations to tackle under- and over-segmentation issues.
2023; Stanić et al., 2023; Lachapelle et al., 2023; Kirilenko
et al., 2024; Gopalakrishnan et al., 2024; Wiedemer et al., 2024; Didolkar et al., 2024; Mansouri et al., 2024; Kori et al., 2024a;b; Jung et al., 2024; Fan et al., 2024) extend the slot structure from various aspects to improve the object segmen- tation performance. Although achieving excellent results, they often fail to scale to complex real-world images as in- vestigated in (Yang& & Yang, 2022; 2024). To overcome this limitation, a line of works (Weis et al., 2021) use additional information such as motion and depth to identify objects. Unfortunately, this precludes learning on most real-world images, which do not have motion or depth information.
3. unMORE
Object-centric Learning with Pretrained Features: Very recently, with the advancement of self-supervised learn- ing techniques, strong object semantic and localiza- tion hints emerge from these features, like DINO/v2 (Caron et al., 2021; Oquab et al., 2023) pretrained on Ima- geNet (Deng et al., 2009) without any annotation, An in- creasing number of methods leverage such features for unsu- pervised salient/single object detection (Voynov et al., 2021; Shin et al., 2022a; Tian et al., 2024), or multi-object seg- mentation (Siméoni et al., 2024), or video object segmenta- tion (Aydemir et al., 2023; Zadaianchuk et al., 2024). Rep- resentative works include the early LOST (Siméoni et al., 2021), ODIN (Hénaff et al., 2022), TokenCut (Wang et al., 2022b), and the recent DINOSAUR (Seitzer et al., 2023), CutLER (Wang et al., 2023a), and UnSAM (Wang et al., 2024). These methods and their variants (Wang al., 2022a; Singh et al., 2022; Ishtiak et al., 2023; Wang et al., 2023c;b; Niu et al., 2024; Zhang et al., 2024) achieve very promis- ing object segmentation results on challenging real-world datasets, demonstrating the value of pretrained features. However, they still fail to discover a satisfactory number of objects, and the estimated object bounding boxes and masks often suffer from segnentttmnnnnnoonnn issues. Essentially, this is because these methods tend to simply group pixels with similar features (obtained from pretrained models) as a single object, lacking the ability to discern boundaries be- tween objects. As a consequence, for example, they usually group two chairs nearby into just one object. By contrast, our introduced three level object-centric representations are designed to jointly retain unique and explicit objectness fea- tures for each pixel, i.e., how far away to the object boundary and in what direction to the object center.
3.1. Preliminary
Our objectness network aims to learn three levels of object-centric representations from the large-scale ImageNet dataset. Thanks to the advanced self-supervised learning techniques, which give us semantic and location information of objects in pretrained models, we opt to use pretrained features to extract object regions on ImageNet to bootstrap our objectness network.
In particular, we exactly follow the VoteCut method pro- posed in CuVI.ER (Arica et al., 2024) to obtain a single ob- ject mask (binary) on each image of ImageNet. First, each image of ImageNet is fed into the self-supervised pretrained DINO/v2, obtaining patch features. Second, an affinity ma- trix is constructed based on the similarity of patch features, followed by Normalized Cut (Shi & Malik, 2000) obtain multiple object masks. Third, the most salient mask of each image is selected as the rough foreground object. For more details, refer to CuVLER. These rough masks will be used to learn our object-centric representations in Section 3.2.
3.2. Objectness Network
With single object images and the prepared (rough) masks on ImageNet (the object image denoted as I E R xW x3 object mask as M E R R'Ixx1)), the key to train our ob- jectness network is the definitions of three levels of object- centric representations, which are elaborated as follows.
Object Existence Score: For image I, its object ex- istence score f is simply defined as 1 (positive sample) if it contains valid object, i.e., sum(M) >= 1, and 0 otherwise (negative sample). In the preliminary stage of processing ImageNet, since every image contains a valid object, we then create a twin negative sample by cropping the largest rectangle on background pixels excluding the tightest object bounding box. As illustrated in Figure 2 (a), image #1 is an original sample from ImageNet, whereas image #2 is a twin negative sample created by us.
Object-centric Representations: To represent objects for downstream tasks such as detection, segmentation, matching, and reconstruction, various properties can be used, including object center/centroid, object binary mask (Cai& & Vasconcelos, 2018; Cheng et al., 2022), and object boundary (Park et al., 2019). For example, prior works
3

---

unMORE: Unsupervised Multi-Object Segmentation via Center- Reasoning
Object Center Field: For image I with a valid object mask M inside, its object center field f is designed to indicate the position/center of the object, i.e., the tightest object bounding box center. As illustrated in Figure 2(b), each pixel within the object mask is assigned with a unit vector pointing to the object center [Ch, Cwl, and pixels outside the mask are assigned zero vectors. Formally, the center field value at the (h, whth pixel, denoted as wi), is defined as follows, where f E R Basically, this center field aims to capture the relative position of an object with respect to the pixels of an image.
distance field, denoted as l'b.w) is defined as follows:
maz (S+M)² if M (b,,e)
9 Shhw
=1
f(h,w)
(3)
otherwise
minn (5.(I-M))
where represents element wiie multiplication and fo xx Figure 2(c) shows an example of an object im- age and its final boundary distance field. Our above def- inition of the boundary distance field has nice property that the maximum signed distance value ((h,)) can be eas- ily recovered based on the norm of the gradient of fd at any pixel inside of object as follows. This property is cru- cial to quickly search for object boundaries at the stage of multi-object reasoning as discussed in Section 3.3.
[ C=. [1 C.llulll [0,0],
if M (h, =1
(1)
otherwise
(6.w0) = 1/1l1 af (n.www
of dw
We notice that prior works use Hough Transform to trans- form pixels/points to object centroids for 2D/3D object de- tection (Gall et al., 2011: Qi et al., 2019), which requires learning both directions and distances to object centers. However, our object center field is just defined as unit di- rections pointing to object centers, as we only need to learn such directions to identify multi-center proposals instead of recovering object masks as detailed in Step #2 of See 3.3.
if ff f > 0
Oh
(4)
Notably, the concept of the boundary distance field (Park et al., 2019; Xie et al., 2022) is successfully used for shape reconstruction. Here, we demonstrate its effectiveness for object discovery.
Overall, for all original images of ImageNet, three levels of object-centric representations are clearly defined based on the generated rough object masks in Section 3.1. We also create twin negative images with zero existence scores.
Object Boundary Distance Field: For the same image I and its object mask M, this boundary distance field f is designed to indicate the shortest distance from each pixel to the object boundary. To discriminate whether a pixel is inside or outside of an object, we first compute the sim- ple signed distance field, where the distance values inside the object mask are assigned be positive, those outside negative, and boundary pixels zeros. This signed distance field is denoted as S E RXXWX for the whole im- age, and its value at the (h, w)th pixel S(p,.)) is calculated as follows:
Objectness Network Architecture and Training: Having the defined representations on images, we simply choose two commonly-use existing networks in parallel as our objectness network, particularly, using ResNet50 (He et al., 2016) as a binary classifier to predict object existence scores fc, and using DPT-large (Ranftl et al., 2021) followed by two CNN-based heads to predict object center field f and object boundary distance field 8 respectively. To train the whole model, the cross- cross-entropy loss is applied for learning existence scores, 1.2 loss for the center field, and L1 loss for the boundary distance field. Our total loss is defined as follows and more details are provided in Appendix A.2.
[[ w) - [h. W|]|, -1[[,, w] [h, alll, otherwise
if M (h) =1
S(h,w))
(2)
where the location (T,, w)) is the nearest pixel position on the object boundary corresponding to the pixel (h, w). Detailed steps of calculation are in Appendix A. These signed distance values are measured by the number of pixels and could vary significantly across images with differently-sized objects. Notably, the maximum signed distance value within an object mask M, assuming it appears at the (h, wi)a pixel location, i.e., S(h,,) maz(S S M) ). indicates the object size. The higher 5 ((. the more likely the object is larger or its innermost pixel is further away from the boundary.
f= (=CE(f + 62 +
( ( {r)
(5)
3.3. Multi-Ohject Reasoning Module
With the objectness network well-trained on ImageNet, our ultimate goal is to identify as many objects as possible on complex scene images without needing human labels for supervision. Given single scene image, a naive solution is to endlessly crop many patches with different resolutions at different locations, and then feed them into our pretrained objectness network to verify each patch's 's objectness. How- ever, this is inefficient and infeasible in practice. To this end, we introduce a network-free multi-object reasoning module consisting of the following steps.
To stabilize the training process, we opt to normalize signed distance values as our object boundary distances. Notably, signed distances for foreground and background are normal- ized separately. For the (h, w th pixel, our object boundary
4

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundee Reasoning
10
Split
upper
0,0
Kereel &
lower
-L. left
Scone Image
Proposal
Center Field fy
Anti-center Map fg
right
Figure 3: An illustration of kernel-basek operation for multi-center detection and proposal splitting.
Step #0 Initial Object Proposal Generation: Given a scene image I E RMx x33, we randomly and uniformly initialize total of T bounding box proposals by selecting set of anchor pixels on the entire image. At each an- chor pixel, multiple sizes and aspect ratios are chosen to create initial bounding boxes. More details are provided in Appendix A.3. For each proposal P. its top-left and bottom-right corner positions in the original scene image will always be tracked and denoted as Pm, P"2, J. We also linearly scale up or down all proposals to have the same resolution of 128 x 128 to feed into our objectness network subsequently.
Tº this proposal P is likely to have 2 2 crowded objects and will be split at the corresponding pixel location with the highest value. As shown in the right block of Figure 3, we safely split the proposal P into 4 subproposals at the highest anti-center value (yellow star): {left, right, upper, lower} halves. Each subproposal is regarded as a brand-new one and will be evaluated from Step #1 again. With this design, the particularly challenging under-seggmentation issue often incurred by multiple crowded objects can be resolved.
2) If the highest value of fp is smaller than the threshold TO, the proposal P is likely to have just one object, multiple objects but they are far away from each other, i.e., more than 5 pixels apart. In this regard, we simply adopt the connected- component method used in CuVLER (Arica et al., 2024) to split the proposal P into subproposals. Specifically, for its center field f all pixels that are spatially connected and have non-zero unit vectors are grouped into one subproposal. Each subproposal is regarded as a brand-new one and will be evaluated from Step #1 again.
Step #1 Existence Checking: For each bounding box pro- posal P. we feed the corresponding image patch (cropped from Z) into our pretrained and frozen objectness network, querying its existence score fg. The proposal will be dis- carded if fp is smaller than a threshold r". The higher 7" is, the more aggressive is ignore potential objects.
Step #2- - Center Reasoning: For the proposal P with high enough object existence score, we then query its center field f. from our objectness network. This step #2 aims to evaluate whether fp has only one center or 2 2 centers. If there is just one center, the non-zero center field vectors of f are likely pointing a common position. Other- wise, those vectors are likely pointing multi-positions, In the latter case, the proposal P needs to be safely split into subproposals at pixels whose center field vectors are facing opposite directions. Thanks to this nice property, we propose the following simple kernel- based operation for multi-center detection and proposal splitting.
Step #3 Boundary Reasoning: At this step, the pro- posal P is likely to have a single object, and we query its boundary distance field f from our objectness network. The ultimate goal of this step is to correctly update this proposal's location and size, i.e., the two corner positions [pm p" 17,,,,, p= in its original scene image I, such that the proposal can converge to a tight bounding box of the object inside. Recall that, in Equations 3&4, our defini- tion of the boundary distance field and its gradient have a crucial property. Particularly, the value at a specific pixel of the boundary distance field from the nearest object's boundaries. This means that we can directly use f help update the two corner positions.
fo indicates how far it is away
As shown in the left block of Figure 3, given the center field S 1228x12822 of proposal P. we predefine kernel K E R 5 where each of the (5 5) vectors has unit length and points outward against the kernel center. Details of kernel values are in Appendix A.3. By applying this ker- nel on top of fG with a stride of X 1 and zero- parddngg,, we obtain an anti-center map, denoted as foc E 1218x 128xx1 The higher the anti-center value at a specific pixel, the more likely that pixel is in between multiple crowded objects. Otherwise, that pixel is more likely to be near object center or belongs to the background. Clearly, the former case is more likely to incur under-segmentatiin
Intuitively, if the proposal P has an incomplete object, its borders need to expand. If it has many background pixels, its borders need to contract. With this insight, we only need to focus on boundary distance values of the four borders of f to decide the margins to expand or contract. To this end, we introduce the following border-based reasoning algorithm update [Pm) p puz, p'2].
As illustrated in Figure 4, for the boundary distance field fh E 1228x128X1 a proposal P. we first collect values at four borders (topmost row, leftmost colunn, bottonmost row rightmost column) highlighted by red dotted lines, denoted
For this anti-center map for of the proposal P, 1) if its highest value among all pixels is greater than a threshold
by four vectors: (f)) f f z } E R228 Each of the
5

---

unMORE: Unsupervised Multi-Object Segmentation via Penter-Boundant Reasoning
P
R
P2
Scene Image I
Proposal P
Boundary Distance Field f3
Figure 4: An illustration of border-basee reasoning algorithm to update proposals.
four borders of proposal P is updated as follows:
Overall, with the pretrained objectness network in Section 3.2, and the network-free multi-object reasoning module in Section 3.3, our pipeline can discover multiple objects in single scene images without training an additional detector. This pipeline is named as unMOREdisc in experiments.
mox
pd1 + p
ou, = argmazs
(6)
11 the
pd pM max(S% pp))
v,, = argmart
or 2i |I|
Optionally Training a Detector: As shown in Cutl (Wang et al., 2023a) and CuVLER (Arica al., 2024), the discovered objects from scene images can be used pseudo labels to train separate class agnostic detector (CAD) from scratch. We select and weight each discovered object based on its confidence score. Intuitively, the selected objects should have high object existence scores, homogeneous center fields and boundary fields. More details about the pseudo label selection are provided in Appendix A.5.
p22 + p22 + max(f" pmp
(u
argmax
0%% |I|
IIl
0u
marif
p p
(u.0) argmaz
of
Because (max(s p.) max p. max( p.). max( p. could be positive or negative, this results in the four borders of the proposal P expanding or contracting by themselves. As shown in the rightmost block of Figure 4, the proposal P updated from the blue rectangle to the yellow one whose bottom and right borders expand to include more object parts because their maximum boundary distance values are positive, whereas its top and left borders contract to exclude more background pixels because their maximum boundary distance values are negative. As boundary distance values are physically meaningful, each expansion step will not go far outside of the tightest bounding box and each contraction step will not go deep into the tightest bounding box.
Following CuVLER, we also train class agnostic detector using the same network architecture and training strategy based on our own pseudo labels from scratch. Our trained detector is named as unMORE in experiments.
4. Experiments
Datasets: Evaluation of existing unsupervised multi-object segmentation methods is primarily conducted on the chal- lenging COCO validation set (Lin et al., 2014). However, we empirically find that a large number of objects are ac- tually not annotated in validation set. This may not be an issue for evaluating sully-sueervised methods in literature, but likely gives an inaccurate evaluation of unsupervised object discovery. To this end, we further manually augment object annotations of COCO validation set by labelling ad- ditional 197 object categories. It is denoted as COCO* validation set and will be released to the community. Details of the additional annotations are in Appendix A.16. We also evaluate on datasets of COCO20K (Lin et al., 2014), LVIS (Gupta et al., 2019), VOC (Everingham et al., 2010), KITTI (Geiger al., 2012), Object365 (Shao et al., 2019). and OpenImages (Kuznetsova al., 2020).
Among the total four steps, the center- center-boundary-eee boundary-aware rea- soning Steps #2/#3 are crucial and complementary to tackle the core saader//over-sgmmenaatiie
issues. Once the two cor- ners of a proposal P are updated, we will feed the updated proposal into Step #3 until the corners converge to stable values. During this iterative updating stage, we empirically find that it is more efficient to take slightly larger step size for expansion, and smaller step size for contraction. More details are in Appendix A.3. The efficiency of our direct iterative updating is also investigated in Appendix A.18.
Once the size and location of proposal P converge, a valid object discovered. After all proposals are processed in parallel through Steps #1/#2/#3, we collect all bounding boxes and apply the standard NMS filter out duplicate detections. For each final bounding box, we obtain its object mask by taking the union of positive values within its bound- ary distance field and non-zero vectors within its center field. We also compute a confidence score for each object based on its object existence score, center field, and boundary distance field. More details are in Appendix A.4.
Baselines: For an extensive comparison on COCO* valida- tion set, we include the following three groups of methods.
Group 1- Direct Object Discovery w/o Learnable Modules. The following methods directly discover objects from COCO* val set, without involving any trainable modules.
Free proposed in FreeSOLO (Wang et al., 2022a) to discover multi-objects based on DenseCL. features.
6

---

unMORE: Unsupervised Multi-Object Segmentation via Center- Center-Bounddry Reasoning
Table 1: Quantitative results on COCO* val set. "# of pred obj." refers to the average number of predicted objects per image.
Trainable Module|
AP
AP
Apm
AR
AR
AP
AP
Ap
AR AR
# of pred obj.
FreeMask
3.7
0.6
1.3
4.6
4.6
3.1
0.3
0.9
3.5
3.5
3.7
w/o
Learnable Modules
MaskCut (K=3)
6.0
24
2.9
6.7
6.7
2.3
5.8
5.8
1.8
Direct
MaskCut (K=10)
6.2
2.6
2.9
7.2
7.2
5.3
2.0
2.3
6.2
6.2
2.1
Object
VoteCut
10.8
4.9
5.5
11.3
11.3
9.5
4.0
4.6
9.8
9.8
8.9
Discovery
w/
DINOSAUR
Recon SlorAit
2.0
0.2
0.6
4.8
4.8
1.1
0.1
0.3
2.9
2.9
7.0
Learnable
FOUND
Seg. Head
4.4
1.8
2.1
3.6
3.6
1.3
1.5
3.0
3.0
1.0
Modules
unMORE,..... (Ours)
Obj Net
19.1
9.0
10.1
19.6
19.6
17.8
9.5
18.9
18.9
8.2
UnSAM
Detector 4
10.2
6.3
6.4
36.1
50.1
10.2
6.2
6.3
34.1
46.1
332.2
Training Detectors
CuLLER
Detector
26.0
14.2
14.7
37,9
37.9
22.7
11.2
11.8
32.7
32.7
100.0
CuVLER
Detector 2
28.0
14.8
15.5
37.8
37.8
24.4
11.7
12.6
32.1
32.1
99.7
Ohj. Network
unMORE(Our))
32.6
17.2
18.0
40.9
40.9
29.6
14.4
15.5
36,5
36.5
100.0
Detector x
Image
DINO
DINOx2 s14
MuskCar (K-10)
Center Field
unMORE (Ours)
DINO -..--........................... Figure 5: Results on COCO* validation set. For MaskCut and VoteCut, the eigenvectors of the second smallest eigenvalue for their used DINO/v2 features are visualized. For unMORE center and boundary object representations are visualized.
COCO GT
DINDv2 b14
VoteCat
Boundary Distance Field
proposed in CutLER (Wang al., 2023a) to discover multi-objects based on DINO features. The number of cut K is set as both 3 and 10 in its favor.
MaskCut:
Its best setting is to first train detector on pseudo labels generated by VoteCut on ImageNet train set, and then train a new detector on pseudo labels inferred from the trained detector on the COCO train set..
CuVLER:
VoteCut: proposed in CuVLER (Arica al., 2024) to discover multi-object based on DINO/v2 features.
unMORE (Ours): We just train single detector on two groups of pseudo labels: one group from our discovered objects on COCO train set, another from object pseudo labels generated by VoteCut on ImageNet train set.
Group 2 - Direct Object Discovery w/ Learnable Modules. The following methods use learnable modules aid object discovery, but without training any multi-object detector.
DINOSAUR (Seitzer et al., 2023): It discovers multi- objects by learning to reconstruct DINO features.
4.1. Multi-object Segmentation Results COCO*
FOUND (Siméoni et al., 2023): This is a salient object detection method.
Table 1 shows AP/AR scores of all methods at different thresholds for object bounding boxes and masks.
unMORE,ssc (Ours): We discover multi-objects by network-free reasoning through our objectness network.
Results and Analysis of Methods in Group 1: From rows 1-4 of Table 1, we can see that MaskCut and VoteCut which utilize DINO/v2 features can achieve preliminary perfor- mance. The middle block of Figure shows qualitative results of MaskCut and VoteCut together with their used DINO/v2 features for grouping objects. Basically, these baselines mainly rely on grouping similar per-pixel features (obtained from pretrained DINO/v2) as objects, resulting in multiple similar objects being grouped as just one, as shown in Figure 5 where two cabinets are detected as one.
Group 3- Object Segmentation by Training Additional Multi- object Detectors. The following methods discover objects by training additional detectors. We adopt diverse range of settings for each method and report the highest scores from their best setting. A full list of all settings and results are in Appendix A.9. Note that, all final evaluation is conducted on COCO" val set which is completely held out.
CutLER: Its best setting is to train detectors on pseudo labels generated by MaskCut on ImageNet train set. As mentioned in the original paper, its training stage takes 3 rounds where each round uses the detector of the previous round to infer on ImageNet train set as new pseudo labels.
Results and Analysis of Methods in Group 2: From rows 5-7 of Table we can see that our unMORE surpasses DINOSAUR and FOUND which are even inferior to feature similarity based methods in Group 1, meaning that recon- struction may not be a good object-centric grouping strategy and saliency maps may be misaligned with objectness.
UnSAM (Wang et al., 2024): Its best setting is to train detectors on pseudo objects discovered by MaskCut on ImageNet train set for 3 rounds in the same way as Cut- LER. The final detector is used to infer on SA-IB train set. Another Former trained on these pseudo labels.
Regarding our unMORE the right block of Figure visualizes the learned center field and boundary distance
7

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundees Reasoning
MaskCut
VoteCut
unMORE (Ours)
Ground Truth
NE
Figure 6: Qualitative results of Direct Object Discovery w/o CAD on COCO* val set as discussed in Sec 4.1 Groups 1&2.
J.P.Mxcca nerant
UnSAM
CutLER
CuVLER
unMORE (Ours)
J.MorTan
Ground Truth
Figure 7: Qualitative results from trained detectors COCO val set discussed in Sec 4.1 Group 3,
field, which allow us to easily discover individual objects, especially in crowded scenes. This is also verified by quali- tative results presented in Figure To further validate this insight, we separately calculate scores on images with more than 5/10/15 ground truth objects respectively in Table 5 of Appendix A.8. Our method maintains high scores crowded images, whereas baselines collapse. Results on the original COCO val set (fewer annotations) are in Appendix A.10. More analysis is in Appendix A.17.
trained on the large-scale SA- 1B dataset from SAM (Kir- illov et al., 2023). Results on the original COCO validation set (fewer annotations) are provided in Appendix A.10.
4.2. Zero-shot Detection Results
For each method, we select its best- rest-performing detector in Group Sec 4.1 and directly test on another datasets: COCO20K/ LVIS/ VOC/ KITTI/ Object365/ OpenImages. As shown in Table 2 and Figure 8, unMORE achieves the highest accuracy on all datasets across almost all metrics, showing our generalization in zero-shot detection.
Results and Analysis of Methods in Group 3: From rows 8-11 of Table and Figure 7, we can see that: 1) Our method clearly surpasses all methods by large margin and achieves oate-of-thheaat performance. 2) Both CutLER and CuVLER can achieve reasonable results because additional detectors are likely to discover more objects. 3) The latest UnSAM appears to be incapable of identifying objects precisely, al- though has rather high AR score when its detector is
We also note that, though our method achieves good perfor- mance for zero-shot detection on natural images, its capabil- ity is likely restricted by the learned objectness in training data. For data with significant domain gaps (e.g., medical images), object priors from natural images may not apply.
8

---

unMORE: Unsupervised Multi-Object Segmentation via Center- conter-Boundary Reasoning
Table 2: Quantitative results of zero-shot detection. Each method uses its best model in Group 3. Since KITTI VOCA
Object365/ Openlmages datasets do not have ground truth masks, only bounding box metrics are calculated.
COCO20K
LVIS
AP
AP
AP
AR
AR
AP
AP
Ap
AR
AR
AP
AP
AP
AR
AR
AP
AP
AP
AR
AR
UnSAM
6.3
3.2
3.4
29.7
42.5
6.3
3.1
3.3
27.5
38.0
4.4
2.5
2.7
23.1
35.7
4.5
2.8
2.8
22.9
34,2
CuIL.RR
22.4
11.9
12.5
33.1
33.1
19.6
9.2
10.0
27.2
27.2
8.5
3.9
4.5
21.8
21.8
6.7
3.2
18.7
18.7
CuVLER
24.1
12.3
13.1
32.6
32.6
21.1
10.7
27.2
27.2
8.9
4.1
4.7
20.8
7.2
3.4
17.9
17.9
unMORE (Ours)
25.9
35.4
23.6
30.5
10.4
5.0
24.1
21.4
KITTI
VOC
Object365
Open Images
AP
AP
AP
AR.0AAR
AP
APh
AR (0)
AR" ThX
AP
AP
AP
AR
AR
AP
AP
AP
AR
AR
UnSAM
1.9
0.6
0.8
17.0
21.7
5.1
2.3
2.6
38.8
$1.9
9.1
4.9
5.3
30.5
47.9
6.6
3.7
4.0
34,6
48.7
CuILER
20.8
7.4
9.5
28.9
28.9
36.8
19.3
20.2
44.0
44.0
21.7
10.3
11.5
34.2
34.2
17.2
9.5
9.7
29.6
29.6
CIVLER
18.8
5.9
8.0
27.9
27.9
39.4
20.1
21.5
43.7
43.7
21.9
9.4
10.9
32.5
32.5
18.6
11.3
11.4
29.8
29.8
unMORE (Ours)
26.7
12.6
13.7
34.8
34.8
40.4
21.5
22.7
47.4
47.4
24.7
11.0
12.4
35.9
19.0
10.9
11.2
29.5
29.5
UnSAM
CRILER
CaVLER
unMORE (Ours)
Ground Truth
KITTI
VOC
Object36
Openlmages
Figure 8: Qualitative results for zero-shot detection discussed in Sec 4.2.
5. Ablations
or sedee-segmentttioon in spite of high AR score achieved. 3) The commonly used binary mask far from sufficient to retain complex object-centric representations.
We explore various combinations of these representations to train an objectness network, which then discovers objects as pseudo labels for the final detector. Details of ablation settings are in Appendix A.
More ablations on our multi-object reasoning module, the choices of hyperparameters Teon / //Te // / and the data augmentation for objectness network are in Appendix A.
Table 3: Ablation results of different choices for object-
centric representations on COCO* val set.
EaiteneeCate Field Disunce Fidl
Objast
Obios:
Olres Boundayy
Aff A1C A1
ARhS A
APR AP
ARf Agg
6. Conclusion
35.7
30.9
In this paper, we demonstrate that multiple objects can be ac- curately discovered from complex real-world images, with- out needing human annotations in training. This achieved by novel two-stage pipeline comprising an object-centric representation learning stage followed by multi-object rea- soning stage. We explicitly define three levels of object- centric representations to be learned from the large-scale ImageNet without human labels in the first stage. These representations serve as a key enabler for effectively dis- covering multi-objects on complex scene images in the sec- ond stage. Extensive experiments multiple benchmarks demonstrate the state-of-the-an performance of our method in multi-object segmentation. It would be interesting to extend our framework to large-scale 2D image generation, where the large pretrained generative models may further improve the quality of object-centric representations.
32.5
28.3
32.6
With the above ablated versions, each method generates its pseudo labels on COCO train set. Then detector is trained on these labels together with the same pseudo labels of ImageNet train set, exactly following the setting of our full method Group of Sec 4.1.
From Table can see that: 1) The boundary distance field yields the largest performance im- provement, as it retains critical information of representing complex object boundaries, thus effectively helping dis- cover more objects the multi-object reasoning module. 2) Without learning object existence scores and object center fields, the AP score drops, potentially due to false positives
Results Analysis:

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning
Acknowledgments
Didolkar, A., Goyal, A., and Bengio, Y. Cycle Consistency Driven Object Discovery. ICLR, 2024.
This work was supported in part by National Natural Sci- ence Foundation of China under Grant 62271433 12271431, in part by Research Grants Council of Hong Kong under Grants 25207822 & 15225522.
Engelcke, M., Kosiorek, A. R., Jones, O. P.. and Posner, I. GENESIS: Generative Scene Inference and Sampling with Object- Latent Representations. JCLR, 2020.
Engelcke, M., Jones, O. P., and Posner, I. GENESIS-V2: Inferring Unordered Object Representations without Iter- ative Refinement. NeurIPS, 2021.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Eslami, S. M. A., Heess, N., Weber, T., Tassa, Y., Kavukcuoglu, K., and Hinton, G. E. Attend, Infer, Re- peat: Fast Scene Understanding with Generative Models. NIPS, 2016.
References
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. 2010.
Ahn, J., Cho, S., and Kwak, S. Weakly supervised learning of instance segmentation with inter-pixel relations. In CVPR, pp. 2209-22188 2019.
Fan, K., Bai, Z., Xiao, T., He, T., Horn, M., Fu, Y., Lo- catello, F., and Zhang, Z. Adaptive Slot Attention: Object Discovery with Dynamic Slot Number. CVPR, 2024.
Arica, S., Rubin, O., Gershov, S., and Laufer, S. CuVLER: Enhanced Unsupervised Object Discoveries through Ex- haustive Self-Supervised Transformers, CVPR, 2024.
Foo, A., Hsu, W., and Lee, M. L. Multi-Object Repre- sentation Learning via Feature Connectivity and Object- Centric Regularization. NeurIPS, 2023.
Aydemir, G., Xie, W., and Guney, F. Self-supervised object- centric learning for videos. NeurIPS, 2023.
Biza, O., van Steenkiste, S., Sajjadi, S. M., Elsayed, G. F,, Mahendran, A. and Kipf, T. Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames. ICMI., 2023.
Frank, M. C., Braginsky, M., Yurovsky, D., and Marchman, V. A. Wordbank: an open repository for developmental vocabulary data. Journal of Child Language, 2016.
Gall, J. and Lempitsky, V. Class-Specific Hough Forests for Object Detection. CVPR, 2009.
Brady, J., Zimmermann, R. S., Sharma, Y., Schölkopf. B., von Kügelgen, J., and Brendel, W. Provably Learning Object-Centric Representations. ICML, 2023.
Gall, J., Yao, A., Razavi, N., Luc Van, L., and Lempitsky, V. Hough Forests for Object Detection, Tracking, and Action Recognition. TPAMI, 2011.
Burgess, C. P., Matthey, I.., Watters, N., Kabra, R., Hig- gins, I., Botvinick, M., and Lerchner, A. MONet: Un- supervised Scene Decomposition and Representation. arXiv:1901.11390 2019.
Geiger, A., Lenz, P., and Urtasun, R. Are ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
Cai, Z. and Vasconcelos, N. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
Gopalakrishnan, A., Stanié, A., Schmidhuber, J., and Mozer, M. C. Recurrent Unsupervised Object Discovery. arXiv:2405.17283, 2024.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging Properties in Self Self-Supervised Vision Transformers. ICCV, 2021.
Autoencoders for
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Gird- har, R. Masked-attention mask transformer for universal image segmentation. In CVPR, pp. 1290-1299, 2022.
Greff, K., Rasmus, A., Berglund, M., Hao, T. H., Schmid-
huber, J., and Valpola, H. Tagger: Deep unsupervised perceptual grouping. NIPS, 2016.
Crawford, E. and Pineau, J. Spatially Invariant Unsuper- vised Object Detection with Convolutional Neural Net- works. AAAI, 2019.
Greff, K., Van Steenkiste, S., and Schmidhuber, J. Neural Expectation Maximization. NIPS, 2017.
Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L.., Botvinick, M., and Lerchner, A. Multi-object representation learning with iterative variational inference. ICML., 2019.
Deng, J., Dong, W., Socher, R., Li, L.-.., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
10

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundand Reasoning
Greff, K., Herrmann, C., Belletti, F., Fleet, D. J., Kipf, T., Pot, E., Sela, M., Meyer, H., Beyer, I.., Kundu, A., Wu, T., Rebain, D., Stone, A., Laradji, I., Zhong, F,, Duckworth, D., and Liu, H.-t. D. Kubrie: A scalable dataset generator. CVPR, 2022.
Lachapelle, S., Mahajan, D., Mitliagkas, I., and Lacoste- Julien, S. Additive Decoders for Latent Variables Identi- fication and Partesian-Product Extrapolation. NeurIPS, 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common Objects Context. ECCV, 2014.
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. CVPR, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Lin, Z., Wu, Y.-F,, Peri, S. V., Sun, W., Singh, G., Deng, F., Jiang. J., and Ahn, S. SPACE: Unsupervised Object- Oriented Scene Representation via Spatial Attention and Decomposition. ICLR, 2020.
Hénaff, O. Koppula, S., Shelhamer, E., Zoran, D., Jae- gle, A., Zisserman, A., Carreira, J., and Arandjelović, R. Object discovery and representation networks. ECCV, 2022.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-Centric Learning with Slot Attention. NeurlPS, 2020.
Ishtiak, T., En, Q., and Guo, Y. Exemplar-FreeSOLO: En- hancing Unsupervised Instance Segmentation with Exem- plars. CVPR, 2023.
Löwe, S., Lippe, P., Rudolph, M., and Welling, M. Complex- Valued Autoencoders for Object Discovery. TMLR, 2022.
Jia, B., Liu, Y., and Huang, S. Improving Object-Centric Learning with Query Optimization. ICLR, 2023.
Löwe, S., Lippe, P., Locatello, and Welling, M. Rotating Features for Object Discovery. NeurIPS, 2023.
Jung, W., Yoo, J., Ahn, S., and Hong, S. Learning to Com- pose: Improving Object Centrie Learning by Injecting Compositionality. ICLR, 2024.
Ma, T., Yang, X., and Latecki, L. J. Boosting chamfer matching by learning chamfer distance normalization. In ECCV, pp. 450 463. Springer, 2010.
Karazija, L., Laina, I., and Rupprecht, C. ClevrTex: A Rexture-Rich Benchmark for Unsupervised Multi-Object Segmentation. NeurlPS, 2021.
Mansouri, A., Hartford, J., Labs, V., Zhang, Y., and Bengio, Y. Object-centric architectures enable efficient causal representation learning. ICLR, 2024.
Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. ICLR, 2014.
Niu, D., Wang, X., Han, X., Lian, L., Herzig, R., and Darrell, T. Unsupervised Universal Image Segmentation. CVPR, 2024.
Kirilenko, D., Vorobyov, V., Kovalev, A. K., and Panov, A. I. Object-Centric Learning with Slot Mixture Module. ICLR, 2024.
Oquab, M., Darcet, T. Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In ICCV, 2023.
Park, J. J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, S. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. CVPR, 2019.
Kori, A. Locatello, F., Ribeiro, F. D. S., Toni, F., and Glocker, B. Grounded Object Centric Learning. ICLR, 2024a.
Kori, A., Locatello, F, Santhirasekaram, A., Toni, F., Glocker, B., and Ribeiro, D. Identifiable Object- Centric Representation Learning via Probabilistie Slot Attention. wxXiv:2406.0714), 2024b.
Qi, C. R.,, Litany, O., He, K., and Guibas, L. J. Deep Hough Voting for 3D Object Detection in Point Clouds. ICCV, 2019.
Ranftl, R., Bochkovskiy, A., and Koltun, V. Vision trans- formers for dense prediction. In CVPR, 2021.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-TTuet, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Uni- fied image classification, object detection, and visual re- lationship detection at scale. IJCV, 2020.
Ren, S., He, K., Girshick, R., and Sun, J. Faster R-CNN: To- wards Real-time Object Detection with Region Proposal Networks. NIPS, 2015.

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning
Sajjadi, M. S. M., Duckworth, D., Mahendran, A., van Steenkiste, S., Pavetić, F., Lučié, M., Guibas, J., Greff, K., and Kipf, T. Object Scene Representation Trans- former. NeurIPS, 2022.
Wang, X., Girdhar, R., Yu, S. X., and Misra, I. Cut and
Learn for Unsupervised Object Detection and Instance Segmentation. CVPR, 2023a.
Wang, X., Yang, J., and Darrell, T. Segment Anything without Supervision. NeurIPS, 2024.
Seitzer, M., Horn M., Zadaianchuk, A., Zietlow, D., Xiao, T., Simon-Gabriel, C.-J., He, T., Zhang, Z.,, Schölkopf, B., Brox, T., and Locatello, F. Bridging the Gap Real- World Object-Centric Learning. ICLR, 2023.
Wang, Y., Shen, X., Hu, S., Yuan, Y., Crowley, J., and Vaufreydaz, D. Self- Supervipe Transformers for Unsu- pervised Object Discovery using Normalized Cut. CVPR, 2022b.
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, Li, J., and Sun, J. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019.
Wang, Y., Shen, X., Yuan, Y., Du, Y., Li, M., Hu, S. X., Crowley, J. L.,, and Vaufreydaz, D. TokenCut: Segment- ing Objects in Images and Videos With Self-Supervised Transformer and Normalized Cut. TPAMI. 2023b.
Shi, J. and Malik, J. Normalized cuts and image segmenta- tion. TPAMI, 2000.
Shin, G., Albanie, S., and Xie, W. Unsupervised Salient Object Detection with Spectral Cluster Voting. CVPRW, 2022a.
Wang. Z., Shou, M. Z., and Zhang, M. Object-centric Learning with Cyclic Walks between Parts and Whole. NeurIPS, 2023c.
Shin, G., Albanie, S., and Xie, W. Unsupervised salient object detection with spectral cluster voting. In CVPRW, 2022b.
Weis, M. A., Chitta, K., Sharma, Y., Brendel, W., Bethge, M., Geiger. A., and Ecker, A. S. Benchmarking Unsu- pervised Object Representations for Video Sequences. JMLR, 2021.
Siméoni, O., Puy, G., Vo, H. V., Roburin, S., Gidaris, S., Bursuc, A., Pérez, P., Marlet, R., and Ponce, J. Localizing Objects with Supe-Super Transformers and no Labels. BMVC, 2021.
Wiedemer, T., Brady, J., Panfilov, A., Juhos, A., Bethge, M., and Brendel, W. Provable Compositional Generalization for Object-Centric Learning. ICLR, 2024.
Siméoni, O., Sekkat, C., Puy, G., Vobecky, A., Zablocki, É., and Pérez, P. Unsupervised Object Localization: Observ- ing the Background to Discover Objects. CVPR, 2023.
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., and Gir- shick, R. Detectron2. https: httpt//tttttb//t/t/ com/ sccebokkessarrchhheettttttt.
2019.
Siméoni, O., Zablocki, É., Gidaris, S., Puy, G., and Pérez, P. Unsupervised Object Localization in the Era of Self- Supervised ViTs: A Survey. IJCV, 2024.
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., Sitzmann, V., and Srid- har, S. Neural Fields in Visual Computing and Beyond, Computer Graphics Forum, 2022.
Singh, G., Deng, F., and Ahn, S. Illiterate DALL-E Learns to Compose. ICI.R, 2022.
Yang, Y. and Yang, B. Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images. NeurIPS, 2022.
Stanić, A., Gopulakrishnan, A., Irie, K., and Schmidhuber, J. Contrastive Training of Complex- Valued Autoencoders for Object Discovery. NeurlPS, 2023.
Yang. Y. and Yang, B. Benchmarking and Analysis of Un- supervised Object Segmentation from Real-Workh Single Images. IJCV, 2024.
Thanh Nguyen, D. A novel chamfer template matching method using variational mean field. In CVPR, pp. 2425- 2432, 2014.
Yuan, J., Chen, T., Li, B., and Xue, X. Compositional Scene Representation Learning via Reconstruction: A Survey. TPAMI, 2023.
Tian, X., Xu, K., and Lau, R. Unsupervised Salient Instance Detection. CVPR, 2024.
Zadaianchuk, A., Seitzer, M., and Martius, G. Object- centric learning for real-world videos by predicting tem- poral feature similarities. NeurIPS, 2024.
Voynov, A., Morozov, S., and Babenko, A. Object Seg- mentation Without Labels with Large-Scale Generative Models. ICML, 2021.
Zhang, X., Xie, J., Yuan, Y., Mi, M. B., and Tan, R. T. HEAP: Unsupervised Object Discovery and Localization with Contrastive Grouping. AAAI, 2024.
Wang, X., Yu, Z., De Mello, S., Kautz, J., Anandkumar, A., Shen, C., and Alvarez, J. M. FreeSOLO: Learning to Segment Objects without Annotations. CVPR, 2022a.
12

---

unMORE: Unsupervised Multi-Object Segmentation via Center- Reasoning
A. Appendix
the number of channels is 256. There are two prediction heads for the prediction of the object center field and the object boundary distance, respectively.
The appendix includes:
Details for Object-centric Representations. A.1
Table 4: Architecture of prediction heads for object center
Details for Objectness Network. A.2
field and object boundary distance.
Details for Multi-object Reasoning Module. A.3
center field prediction head
boundary field prediction head
type chanacls activation stride
type
channcis activation stridc
Details for Object Mask and Confidence Score, A.4
layer conv Ixl
512
RELU
layer conv Ixl
512
RELL
layer conv 3x3
512
RELU
layer 2conv 3x3
512
1 RELU
Details for Pseudo Label Process. A.5
layer 3 conv Ixl 1024
RELU
layer 3 conv 1xl
1024
RELL
1
layer 4 conv
RELU
Jayer 4 1xl
RELL
Details for Detector Training. A.6
Objectness Network Training Strategy. The object exis-
Details for Datasets. A.7
tence model trained using the Adam optimizer for 100K iterations with a batch size of 64. The learning rate is set to be a constant 0.0001. The object center and boundary models are jointly trained using the Adam optimizer for 50K iterations with batch size of 16. The learning rate starts 0.0001 and is divided by 10 at 10K and 20K iterations.
More Results on COCO* Validation Set. A.8
Details for CAD Training Settings. A.9
Experiment Results on COCO Validation Set. A.10
Details for Ablation Settings. A.11
More Ablation Studies. A.12
Time Consumption and Throughput. A.13
Objectness Network Training Data. We use the ImageNet train set with about 1.28 million images as the training set for the objectness network. For each ImageNet image, its object mask is the most confident mask generated by Vote- Cut proposed in CuVLER (Arica al., 2024). For the training of the object existence model, negative samples that do not contain objects are created by cropping the largest rectangle region on the background. For positive samples that contain objects, we apply the random crop augmenta- tion onto the original ImageNet image and discard the crop without a foreground object. For the training of the object center and boundary model, we first calculate the ground truth center field and boundary distance field based on the original full ImageNet image. Then, we apply the random crop augmentation onto the original image as well as the two representations. Specifically, the scale of the random crop is between 0.08 to I, which implies the lower and up- per bounds for the random area of the crop. The aspect ratio range of the random crop is between 0.75 and 1.33, Lastly, each image is resized to 128 x 128 before feeding into Objectness Network.
Failure Cases. A.14
More Qualitative Results. A.15
Details of COCO* Validation Set. A.16
More Results and Analysis of Object-centric Representa- tions. A.17
Number of Iterations for Proposal Optimization. A.18
A.1. Details for Object-centric Representations
Calculation of Signed Distance Field. Given bi- nary mask M #xxx we caleulate the dis- tance from each pixel to its closest boundary point with brary DistanceTransform () function in the opencv li- opency. org/4 h////.x/// h/dp////d//// @roup__mgprro www.mmcc.......... html). The function takes a binary mask as input and computes the shortest path length to the nearest zero pixel for all non-zero pixels. Thus, we first compute the distance field within the object, denoted as Soyj, using the object binary mask M. Then, we com- pute the distance field within the background, denoted as Sog. using (1 M). The signed distance field for the whole image is S = Sabj Sby. Specifically, when using di stanccaeeeaaee Trans form (), we set the distance type as L.2 (Euclidean distance) and the mask size be
A.3. Details for Multi-Object Reasoning Module
Initial Object Proposal Generation. Motivated by anchor box generation in Faster R-CNN (Ren al., 2015). We use five scales [32, 64, 128, 256, 512] and three aspect ratios [0.5, 1, 2). At each scale, we randomly and uniformly sam- ple proposal centers based on scale sizes. At each sampled center, we generate three boxes with different aspect ratios.
A.2. Details for Objectness Network
Objectness Network Architecture. The object existence model employs ResNet50 (He et al., 2016) as the backbone. Following this backbone, the classification head consists of single linear layer with output dimension of 1 and sigmoid activation layer. The prediction for the object center field and the object boundary distance shares the same DPT-large (Ranftl et al., 2021) backbone with a 256- dimensional output size. Dense feature maps extracted from this backbone have the same resolution as input images and
Figure 9: Predefined Kernel for Center Reasoning
Predefined Kernel for Center Reasoning. As illustrated
13

---

unMORE: Unsupervised Multi-Object Segmentation via Center- Reasoning
in Figure 9, each position within the kernel is defined as 2-dimensional unit vector pointing towards the center of the kernel. Specifically, the value at the kernel center with position [2, 2] is (0, 0). The value at the (i. position, denoted as Kk. is defined and normalized as:
since objects can still be well seen from a proposal larger than its tightest bounding box. Thus, further adjust the calculated updates with an adjustment ratio Tadjust = 0.5.
Instead of directly using Eq. 6, we use the following formu- las to calculate boundary update:
|2, 2] - [ J = || 2] [i.
maz (f (F
| (f), !Il
To evaluate how Center Field matches with this anti-center pattern, we apply convolution onto Center Field with this kernel to calculate their average cosine similarity for each pixel in the Center Field. We set the threshold To be 0.25.
p p
I|
05% af";
||
Tadjuct
|| Ou : Du
05 af.,
0u
de
where (u, v) argmarf p=
(9)
P pm max (f
||max f%:)
More Details for Center Reasoning. While deriving the anti-center map with the predefined kernel, we also find the boundary of the Center Field. Since on the anti-center map, values at the boundary of the Center Field will also be positive, we thus ignore the values on the Center Field boundary. Examples of center reasoning are provided in Figure 10.
af; of
Tadjust
- de
of 05
du
where (u, v) argmar fb.
(10)
pu2 pud max (f
|max( (f
||
0f%
||
Tadjust
a5z af
dc
do
where (u, v) argmar Po.
(11)
More Details for Boundary Reasoning. Let fo E R R128x128x1 be the distance field for proposal P and V&, E 1282x128x2 is the gradient map for fgr where
max
pr)
| (F),) }
pma
pe
Tadjust
of 8f du
05
85
Vfplu,0 = ( 0f: of²). And | f R128x128x1 is the norm for the gradient map. To make the bounding box update more stable, we use two strategies: (1) Use the eraged distance field gradient to replace the gradient at single pixel position; (2) Apply adjustment on the calculated update step for a more aggressive expansion and conserva- tive contraction.
where (u, v) argmax f
(12)
du
Parameters for Proposal Updating, Each proposal under- goes at most 50 iterations of updates. For efficiency, we stop proposal from being updated once it meets the following criteria. Specifically, the calculated maximum expansion for the proposal should be smaller than 0 (it means the border moves outside of the object boundary), and the maximum shrinkage should be smaller than small margin, which we set be 16 pixels. While it is acceptable for the proposal to be slightly larger than the tightest bounding box, it should not be smaller. Examples of boundary reasoning can be found in Figure 10.
(1) Since the distance field within the object and outside the object are normalized separately, the gradient average op- eration needs be applied separately. Thus, we first apply sigmoid function onto the boundary field to generate mask for foreground o (f)) and background 1 - -ff)) Then gra- dients are averaged separately on the two masks and com- bined as the averaged gradient norm map for the distance field AV G[I f(ll)) E R 188 128x1, We replace || f[I) with AVG(I (D))
A.4. Details for Object Mask and Confidence Score Calculation
when calculating box updates.
For converged proposal P, we can compute its object mask M, p as the union of mask from center field and mask from boundary field:
Ed(s)) 1VS
AVG(IVS)I)
o(F))
Eo(F))
Σ( -(55))) ||VS
(11 (f))))
1,
if || 2 0.5 otherwise
E(f-o(f))
M Meentr p
(13)
0,
(2) Empirically, box contraction needs to be more conser- vative since objects could be overlooked if the proposal is over-tightened. For example, for person wearing a tie, if the proposal around the person gets shrunk too much, the object of interest may transfer to the tie instead. Also, for efficiency, it is suitable to make more aggressive expansion
1.
if (o()))) 2 0.5 otherwise
M boundary
(14)
0,
M, = U( Mboundary)
(15)
14

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundann Reasoning
To caleulate the confidence score confp for proposal P, consider its object existence score, center field, and boundary field. Specifically, we also consider mask area when calculating the confidence by comparing the object area in with other objects' areas within the same image. Suppose there are K discovered objects within the image, the final score calculated as:
version of COCO, this dataset is generally used to evaluate unsupervised approaches.
LVIS (Gupta et al., 2019): LVIS (Large Vocabulary Instance Segmentation) is dataset for long tail instance segmenta- tion. It contains 164,000 images with more than 1,200 cate- gories and more than 2 million high-quality instance-level segmentation masks.
M MARKEK Σ M (16)
consp - [g (fl)m
KITTI (Geiger et al., 2012): KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is one of the most popular datasets for use in mobile robotics and autonomous driving. Our method is evaluated with 7521 images from its trainval split.
A.5. Details for Pseudo Label Processing
Given a set of discovered objects from scene images, we perform selection and assign each of them a weight to use them as pseudo labels for training the detector. Following the definition in the Section A.4, an object proposal P will be selected if it satisfies three conditions below:
PASCAL VOC (Everingham et al., 2010): The PASCAL Visual Object Classes (VOC) 2012 dataset is widely used benchmark for object detection, containing 1464 training images and 1449 validation images.
Object365 V2 (Shao et al., 2019): Objects365 is a large- scale object detection dataset. It has 365 object categories and over 600K training images. We evaluate our method in terms of object detection on its validation split with 80K images.
fp 2 F: max (II)) 2 Feonf: mar(f)) 2 Foon (17)
The three thresholds correspond to object existence score (Femg)) maximum norm in center field (Temm)) and maxi- mum value in boundary distance field (r& (Foong). In our paper, we set:
Openlmages V6 (Kuznetsova et al., 2020): OpenImages V6 is large-scale dataset, consists of 9 million training images, 41, 41,620 validation samples, and 125,456 test samples. We evaluate our method in terms of object detection on its validation split.
Teonf 0.5; Tcoms 0.8;
_ Toom = 0.75
(18)
For each selected proposal, its weight for the detector train- ing is determined by its relative area in the scene image:
A.8. More Results on COCO Validation Set
0.25
Σ M, MARKER Σ M
We present a detailed evaluation on COCO* validation dataset based on object count in Table 5. We can see that, when the number of objects in each image is rather small (e-g.. [0 [0-4]). the results of nop-performing baselines Vote- Cut/CuLLER are comparable to our method, all yielding high scores. However, as the number of objects per image increases (e.g., 2 5 objects), our unMOREaise! unMORE consistently outperforms all baselines by growing margins, demonstrating the superiority of our method in dealing with challenging crowded images.
A.6. Details for Detector Training
The architecture for the Class Agnostic Detector is Cascade Mask RCNN. All experiments are performed with the De- tectron2 (Wu et al., 2019) platform. Detectors are optimized for 25K iterations using SGD optimizer with a learning rate of 0.005 and a batch size of 16. We use a weight decay of 0.00005 and 0.9 momentum. Following CutLER (Wang et al., 2023a), we also use copy-paste augmentation with a uniformly sampled downsample ratio between 0.3 and 1.0.
Notably, UnSAM achieves high /AR scores (used in the original UnSAM paper to measure the aver- age recall rate without limiting the number of predictions), but its / mask scores (only considers the top 100 100 predictions per image and commonly adopted for object segmentation) are clearly lower. This is because UnSAM focuses on excessively partitioning images by clustering granular segments, which sacrifices the accuracy of object discovery, but tends oversegment objects. This is also qualitatively validated in the Figure 7 and Figure 8 in the main paper.
A.7. Details for Datasets
COCO (Lin et al., 2014): The MS COCO (Microsoft Com- mon Objects in Context) dataset a large-scale object de- tection and segmentation dataset. The COCO in the paper refers to the 2017 version that contains 118K training im- ages and 5K validation images.
COCO 20K (Lin et al., 2014): COCO 20K is a subset of the COCO trainval2014 with 19817 images. Since it contains images from both training and validation set from the 2014
15

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundan Reasoning
Table 5: Quantitative results on COCO* validation set based on object count. "# of GT obj." refers to the average number of ground truth objects per image, while "# of pred obj." refers to that of predicted objects.
Ao ubj.
Dirret Object Disowery
Triniing Deteciees
AP
AP
AR
AP
AP1:
A.
ARg0
pecd.bb
AP5:
AP1:
AP
AP'
AF1
AP
ARRc
AR
peed
MaskCir (K=1)
25.1
12.1
13.3
28.5
28.5
22.5
5.9
10.6
24.2
24.2
1.8
UsSAM
15.5
10.5
10.2
66.4
73.3
15.9
10.4
10.1
60.1
68.5
244.1
[0.4]
MuskCur (K-ii))
34.5
11.7
129
20.3
22.
21.
8.8
10.3
24.8
248
1.9
CaLER
55.2
35.4
34.4
61.2
61.2
51.2
20.0
28.6
52.4
52.4
100.0
NoreCui
39.1
37.0
19.0
COVLER
63.5
60.5
536
52.6
99.9
u (Ours)
21.2
38.2
38.7
42.1
22.8
37.8
MEMORE (Ours)
23.7
23.1
59.6
52.9
29.5
52.8
52.8
100,0
MiskCar (K-3)
10.7
48
53
10.9
10.3
9.
6.3
9.0
9.6
1.9
USSAM
12.3
8.6
8.7
299
133
8.3
'6.22
36.5
[5.9]
MaskCur (K=10)
VoteCin
11.4
17.2
7.6
500
8.6
5.5
11.4
17.5
11.4
17.5
15.4
9.6
6.7
3.6
4.4
7.4
15.0
9.9
1500
9.9
9.0
2.1
CaNLER
COILER
57.7
50.0
21.1
21.4
21.7
21.9
49.3
48.4
49.3
4%.4
33.3
94.1
16.7
16.6
17.5
17.
41.3
42.5
42.5
41.3
100,0
99.6
unMORK
25.8
12.8
HIMORE (Ques)
MankCar 3
37
23"
15
USSAMM
11.2
8.7
565
SET
11.3
'67
"58.6"
"ST8"T"
[10,14]
MaskCar (K-10)
VotCCul
8.8
5.4
2.4
3.0
3.9
2.7
9.5
5.5
5.5
9.5
4.7
7.2
3.6
1.4
3.3
1.9
4.8
K.1
4.8
8.1
9.2
2.3
CUVLER
CHIFER
28.5
26.3
13.7
13.2
15.2
14.3
39.7
403
40.1
39.7
22.8
24.9
11.1
10.2
11.5
12.4
34.0
34.9
34.0
34.0
100,0
99.7
a (Ours)
82
866
1N.5
MEMORE (Qurs)
30.5
38.6
MaskCat (K-1)
0.5
19
19
16
1.6
166
UsSAM
8.9
5.2
348
40.9
8.6
4.
24.2
38.1
475.7
(15.4)
MaskCar (K-10)
1.7
0.5
03
211
21
1.5
0.4
0.2
1.9
1.9
2.8
CUILER
19.3
9.0
10.0
2901
29.0
15.5
6.7
7.6
25.1
25.1
100.0
VoteCut
4.2
1.4
188
4.6
4.6
32
1.2
144
4.0
4.0
9.3
CUVLER
21.4
9.7
10.9
28.4
28.4
17.0
7.1
8.3
24.4
24.4
99.6
mnMORE (Ours)
65
14,
1400
123
13,4
13,
MIMORE (Ours)
29,0
14.2
3344
33.4
12.5
30,0
100,9
A.9. Details for CAD Training Settings
Setting #2: It is trained two groups of pseudo labels:
one group from its discovered objects on COCO train set, another from object pseudo labels generated by VoteCut on ImageNet train set.
In Sec 4.1 Group 3, since four methods train CAD with dif- ferent settings, we adopt a diverse range of training settings, which are detailed as follows. The best setting for each method is marked with bold. Full results for all settings on COCO* validation set are in Table 6.
Setting #3: It is trained only on object pseudo labels generated by VoteCut on ImageNet train set.
Setting #4: It first uses the detector of Setting #3 to infer object pscudo labels on COCO train set, and then trains a new detector on these pseudo labels.
1) For UnSAM, it has two detectors trained under two set- tings below. Both models are from the original paper and included for reference.
4) For our method, named umMORE, we train two separate detectors under two settings:
Setting #1: It trains a detector on pscudo objects discov- ered by MaskCut on ImageNet train set, and then the detector used infer scene images jointly with Mask- Cut.
Setting #1: It is trained only on pseudo objects discovered by our method on COCO train set.
Setting #2: The detector trained in its Setting #1 is used to infer pseudo objects on SA-1B train set. Another Fask2Former is trained on these pseudo labels for in- ference on scene images.
Setting #2: is trained on two groups of pseudo labels: one group from our discovered objects on COCO train set, another from object pseudo labels generated by VoteCut on ImageNet train set.
2) For CutLER, it has three detectors trained under three settings below. The Settings #1/#2 are fairly comparable with our Settings #1/#2, whereas its Setting #3 is from the original paper.
A.10. Results on the Original COCO Validation Set
This section presents the experiment results evaluated on original COCO validation set. Table 7 shows the quantitative results on COCO validation set. Table 8 shows quantitative results of detectors with different settings on the original COCO validation set.
Setting #1: It is trained on pscudo objects discovered by its own MaskCut on COCO train set.
Setting #2: It is trained on two groups of pseudo labels: one group from its discovered objects COCO train set, another from object pseudo labels generated by MaskCut on IeageNet train set.
A.11. Details for Ablation Settings
As mentioned in Sec 5, We explore various combinations of these representations to train objectness network, which then discovers objects as pseudo labels for the final detector. Details of ablation settings are as follows:
Setting #3: It is trained on object pseudo labels generated by MaskCut ImageNet train set.
3) For CuVLER, it has four detectors trained under four settings below. The Settings #1/#2 are fairly comparable with our Settings # #1/#2, whereas its Settings #3/#4 are from the original paper.
1) Only using binary mask the object-centric repre- sentation: In the task of object segmentation, binary mask is probably the most commonly-used object representation. In particular, we remove all of our three object-centric rep- resentations, but just train the same objectness network to predict binary mask. Then, when discovering multi-
Setting #1: It is trained only on pseudo objects discovered by its own VoteCut on COCO train set.
16

---

unMORE:
Unsupervised Multi-Object Segmentation via Center- Center-Boundary Reasoning
Table 6: Quantitative results of detectors with different settings
COCO* validation set.
Training Settings
AP
AP7s
Aphia
AR box
30
AP
Apmak
mack
UnSAM
Setting #1
3.5
2.1
2.3
30.5
3.2
2.0
2.1
27.2
Setting #2
10.2
6.3
6.4
36.1
10.2
6.2
6.3
34.1
CutL.RR
Setting
21.2
10.8
11.6
33.4
18.2
27.7
Setting #2
23.6
11.8
12.6
33.7
19.8
8.3
9.5
28.4
Setting #3
26.0
14.2
14.7
37.9
22.7
11.2
11.8
32.7
CuVLER
Setting
26.1
13.2
14.1
36.0
22.6
10.3
11.3
30.6
Setting #2
27.0
13.0
14.2
35.0
23.2
10.1
11.4
29.8
Setting #3
27.2
14.0
14.9
37.2
23.2
10.7
11.8
30.2
Setting #4
28.0
14.8
15.5
37.8
24.4
11.7
12.6
32.1
unMORE (Ours)
Setting
31.2
15.6
16.8
40.0
28.8
12.7
14.9
36.1
Setting
32.6
17.2
18.0
40.9
29.6
14.4
15.5
36.5
Table 7: Quantitative results the original COCO validation dataset.
Trainable Module
AP
AP
Ap
AR
AR
Ap AP
Apr
AR
AR
avg. # obj.
PreeMask
4.1
0.7
1.4
4.3
4.3
3.5
0.4
1.1
3,4
3.4
3.7
w/o
MuskCut (K=3)
7.7
1.8
2.3
1.8
Learnable
Direct
MaskCut (K=10)
3.1
8.2
8.2
5.5
1.7
2.2
6.9
6.9
2.1
Modules
Object
VoteCut
11.0
5.0
5.6
12.4
12.4
9.4
4,0
4.6
10.5
10.5
8.9
Discovery w/o
DINOSAUR
Recon. SlotAn
2.1
0.2
0.6
5.5
5.5
0.8
0.1
0.2
2.5
2.5
7.0
Leurnable
FOUND
Seg. Head
4.7
2.1
2.3
4.5
4.5
3.7
1.5
1.8
3.7
3.7
1.0
Modules
unMORE (Ours)
Obj. Net
15.7
7.9
16.5
16.5
14.7
6.9
7.5
15.9
15.9
8.2
UnSAM
Detector x
3,4
42,4
3.3
37.9
332.2
Training
CutLER
Detector x
22.9
11.7
12.4
31.8
31.8
7.3
8.8
23.9
23.9
100.0
Detectors
CuVLER
Detector
23.4
12.1
12.8
32.2
32.2
20.4
9.6
10.4
26.8
26.8
99.7
Obj. Network
unMORE (Ours)
25.4
12.7
13.6
35.2
10.7
11.7
30.3
30.3
100.0
Detector x
objects scene images, manually set suitable step size to extensively search object candidates by querying the pretrained network.
8) Our full three-level object-centric representations: This is our full framework for reference.
A.12. More Ablations
2) Only using a binary mask and an object existence score: This is to evaluate whether the object existence score can be useful for better object segmentation. In the absence of object boundary field, the binary mask representation can update bounding boxes.
Selection of Fixed Step Size for Binary Baseline. Since the information provided by binary mask representation is very limited, the final discovered objects can be very sensitive to the step size. In order to choose a good step size in favor the binary mask baseline, we randomly select 100 images from COCO* validation set and evaluate the results for a step size of 5, 15, 20, 30. According to the results shown in Table 9, select 20 the fixed step size.
3) Only using binary mask and an object center field: This is to evaluate whether the object center field can be useful for better object segmentation. In the absence of object boundary field, the binary mask representation can update bounding boxes. 4) Using binary mask, object existence score and center field: This is to evaluate whether both object exis- tence score and center field can be useful for better object segmentation. In the absence of object boundary field, the binary mask representation can update bounding boxes.
Ablation on Parameters for Pseudo Label Processing,
We perform ablation studies on the parameters used in A.5, Specifically, we choose wide range, iee., (0 ~ 0.95) for score thresholds of object existence Temm object center T and object boundary -oomf on 7 datasets. As shown in Tables 10&11, more tolerant thresholds lead to higher AR scores because more objects can be discovered, but a decrease in AP because of low-quality detections. On the other hand, if thresholds are too strict, both AR and AP scores drop because only limited number of objects are discovered. Nevertheless, sensitive the selection of thresholds as it demonstrates good performance across different thresholds.
5) Only using an object boundary field: This is verify the importance of object boundary field.
6) Only using object boundary field and existence score: This is to evaluate whether adding the existence score can help object segmentation on top of the object boundary field.
method is not particularly
7) Only using an object boundary field and center field: This is evaluate whether adding the center field help object segmentation top of the object boundary field.
Ablation on Random Cropping Augmentation for the Objectness Network. During training our objectness net- work on ImageNet, we originally apply random cropping
17

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boou cetter-ouundary Reasoning
Table 8: Quantitative results of detectors with different settings on the original COCO validation set.
Training Settings
AP
AP7s
Aph
AR hos
3P
AP
Ap
A
maak
UnSAM
Setting #1
2.1
1.1
1.2
27.0
1.8
0.9
1.0
23.5
Setting #2
5.9
3.2
3,4
30.0
5.9
3.1
3.3
27.4
CutLER
Setting
19.3
9.9
10.6
16.3
7.3
23.2
Setting #2
20.8
10.4
11.1
29.7
17.2
7.0
8.1
23.3
Setting #3
21.9
11.8
12.3
32.7
18.9
9.2
9.7
27.0
CuVLER
Setting #1
22.9
11.7
12.4
31.8
18.7
7.3
8.8
23.9
Setting #2
11.3
12.3
31.2
19.7
8.5
24.9
Setting #3
22.9
11.8
12.6
32.9
19.3
8.9
9.8
25.1
Setting #4
23.4
12.1
12.8
32.2
20.4
9.6
10.4
26.8
unMORE (Ours)
Setting #1
24.1
11.2
12.5
34.2
22.2
9.9
11.1
29.9
Setting
25.4
12.7
13.6
35.2
22.9
10.7
11.7
30.3
Table 9: Results of different step sizes for binary baseline COCO* validation set.
step size
APhox
Aphox
Apbox
AR
Apamask
A
Apmask
ARmask
10.4
4.1
4.9
10.8
9.4
3.7
4.3
9.6
1.5
12.4
5.8
6.4
12.1
10.7
4.8
5.4
10.7
13.1
6.3
12.1
11.6
5.3
5.9
10.7
30
11.8
5.4
6.1
11.5
10.1
4.6
5.1
10.1
augmentation. Here, we conduct an additional ablation study by omitting the random cropping operation during training the objectness network while keeping all other set- tings the same. Table 12 shows the quantitative results the COCO* validation set. We can see that random cropping is indeed helpful for the objectness network to learn robust center and boundary fields. Primarily, this is because during multi-object reasoning stage, many proposals just have partial or fragmented objects, but the random cropping aug- mentation inherently enables the objectness network to infer rather accurate center and boundary field for those partial objects, thus driving the proposals to be updated correctly.
A.13. Time Consumption and Throughput
Time consumption is summarized in Table 14. unMORE takes 10 hours to train the objectness network and is slower for Direct Object Discovery. However, tector unMORE requires only 30 hours to train, benefit- ing from the high-quality pseudo labels from unMOREaises while baseline detectors take over 60 hours. Ultimately, the inference speed of our unMORE matches that of CutLERR and CuVLER.
subsequent de-
Regarding the throughput, for each image on average, the number of initial proposals is 1 1122.7, whereas the number of predicted objects from unMORE is 8.9. Most initial proposals have low existence scores and are discarded at the first iteration. The Mon-Maximum Suppression (NMS) will also remove redundant proposals.
Ablation on Rough Masks for Training Objectness Net- work. We conducted the following ablation study on four types pseudo-masks.
SelfMask (Shin et al., 2022b): For each image, we em- ploy the strong unsupervised saliency detection model SelfMask to predict a salient region as the pseudo label.
A.14. Failure Cases
We present failure cases in Figure 11 and discuss limitations as follows.
MaskCut: For each image, we use the first object discov- ered by MaskCut the pseudo label.
1. Direct Object Discovery of unMORE takes time. could possible to leverage reinforcement learning tech- niques learn an efficient policy discover objects.
VoteCut: It's used in our main paper.
VoteCut+SAM: For each image, rough mask is gener- ated by VoteCut, and its bounding box is used as prompt for SAM to predict the final pseudo mask. While this yields the best pseudo labels, SAM is fully supervised model, so this ablation is for reference only, As shown in Table 13, our method amenable to all types of rough masks, though their quality affects unMORE,ise performance. While SAM scores highest, its improvement over VoteCut is not substantial, as it still relies on bounding box prompts from VoteCut. Importantly, our method does not depend on specific pretrained features, enabling the use of enhanced pretrained models in the future.
2. Our method struggles separate overlapping objects with similar textures, as shown in the attached Figure 11. Additional language priors may help alleviate this issue.
A.15. More Visualizations
18

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundee Reasoning
Table 10: Ablation results for thresholds of object existence cconf. object center Toon and object boundary Tcons COCO
validation set.
Teon
Toom
AP
Apbox
AR
AP
Apmas
Back
0.0
0.8
0.75
31.2
16.7
17.4
41.0
28.7
14.6
15.3
37.2
0.25
0.8
0.75
31.5
16.7
17.5
40.8
28.6
14.3
15.2
36.7
0.5
0.8
0.75
32.6
17.2
18.0
40.9
29.6
14.4
15.5
36.5
0.75
0.8
0.75
30.8
16.2
16.9
38.9
27.7
13.3
14.3
34.7
0.95
0.8
0.75
28.1
13.4
14.7
34.4
24.3
10.7
12.1
30.1
0.5
0.0
0.75
32.5
16.4
17.5
40.0
29.2
13.6
14.9
35.8
0.5
0.25
0.75
31.8
16.4
17.3
39.9
28.5
13.5
14.7
35.7
0.5
0.5
0.75
31.0
16.2
17.0
40.2
27.7
13.3
14.4
36.0
0.5
0.8
0.75
32.6
17.2
18.0
40.9
29.6
14.4
15.5
36.5
0.5
0.95
0.75
29.8
15.8
16.5
38.1
26.8
13.2
14.1
34.2
0.5
0.8
0.0
31.8
16.0
17.0
38.7
28.4
13.2
14.5
34.6
0.5
0.8
0.25
31.2
16.1
17.0
38.9
27.8
13.2
14.3
34.7
0.5
0.8
0.5
31.7
16.9
17.5
40.6
28.4
13.7
14.7
36.0
0.5
0.8
0.75
32,6
17.2
18.0
40,9
29.6
14.4
15.5
36.5
0.5
0.8
0.95
31.6
17.5
17.9
39.8
28.0
13.3
14.5
35.0
Table 11: Ablation results for thresholds of object existence f object center Toonf and object boundary ofon on COCO20K, LVIS, KITTI, VOC, Object365 and OpenImages.
COCO
COCOZOK
LVIS
Object365
OpenImages
Foom
Teon/
coo
AP
AR
AP
ARtuk
AP
AR
AP
ARtikk
AP
AR
AP
AR
AP
AR
AP
(00
AP
AR km
AF
0.0
0.8
23.8
24.3
31.1
24.9
22.6
46.9
36.3
29.5
0.25
0.8
0.75
24.1
34.8
22.0
30.3
24.6
35.0
22.6
30.6
10.2
24.4
8.7
21.9
25.0
34.0
39.1
46.6
23.8
36.0
18.7
29.4
0.5
35.9
29.5
0.75
0.8
0.75
24.5
33.7
21.9
28.8
25.1
34.1
22.7
29.2
9.9
22.5
8.3
20.0
25.5
33.6
40.4
46.7
23.8
36.0
18.7
29.4
0.95
0.8
0.75
23.2
30.2
19.9
25.0
23.8
30.5
20.6
25.3
8.7
18.8
6.9
16.3
21.6
29.6
39.4
43.7
21.6
30.0
26.5
0.5
0.0
0.75
25.7
34.5
22.8
29,8
26.2
34.8
23.4
30.1
10.4
23.3
8.5
20.9
28.7
35.5
41.3
47.0
24.5
35.1
19.7
29.0
D.5
0.25
0.75
25.0
34.4
22.2
29.5
25.6
34.8
23.0
29.8
10.1
23.2
8.3
20.6
27.7
33.6
41.0
46.8
23.8
35.1
19.3
29.0
D.5
24.5
29.9
25.1
30.1
9.8
23.6
21.1
24.1
32.7
35.3
29.7
0.5
0.8
0.75
25.4
35.2
22.9
30.3
25.9
35.4
23.6
30.5
10.4
24.1
8.9
21.4
26.7
34.8
40.4
47.4
24.7
35,9
19.0
29.5
0.5
0,95
0.75
23.7
32.9
21.1
28.3
24.3
33.2
21.8
28.5
9.6
21.6
8.2
19.3
25.7
33.3
38.6
45.6
22.5
33.2
18.3
28.4
0.5
0.8
24.7
33.4
21.9
28.7
25.3
33.6
22.6
29.0
10.1
22.3
8.2
19.8
27.4
33.4
40.0
45.9
23.6
33.8
19.3
28.3
0.5
0.8
0.25
24.6
33.6
21.8
28.9
25.3
34.0
22.5
29.3
9.8
22.4
8.0
19.8
26.7
33.5
40.7
46.1
23.2
34.1
19.7
28.6
0.5
0.8
25.3
38.2
22.4
30.0
25.9
35.3
23.1
30.4
10.0
23.6
20.9
25.4
34.3
41.3
47.8
23.7
35.8
19.9
29,9
0.5
0.8
0.75
25,4
35.2
22.9
30.3
25.9
35,4
23,6
30.5
10.4
24.1
8.9
21.4
26.7
34.8
40.4
47.4
24.7
35.9
19.0
29.5
0.5
0.8
0.95
20.4
32.2
19.7
28.6
24.4
34.4
22.7
29.8
10.5
23.3
9.0
21.0
29.7
35.1
37.6
46.4
23.8
34.8
17.8
29.2
Table 12: Ablation results on COCO* validation set for random cropping augmentation of the objectness network.
AP
APhex
Aplox
AR
A
Ap
Apmask
AR
unMORE,iic
19.1
9.0
10.1
19.6
17.8
8.7
9.5
18.9
(with random cropping)
unMORE,.is
15.7
7.5
8.2
18.1
15.6
6.6
7.9
17.4
(w/o random cropping)
Table 13: Ablation study for rough masks.
Rough Masks
features/ Supervision
AP
APhs
Aphox
AR
AR
AP
Apmaak
Apiiss
ARmukk
AR
SeltMask
DINO.b16 MoCov2, SWAV
13.2
6.1
4.8
16.4
16.4
12.0
5.0
5.6
15.3
15.3
MaskCut
DINO b8 D8 DINO_s8, DINO
16.3
7.3
6.4
17.7
17.7
14.3
5.7
6.1
18.7
18.7
unMORE.tcc.
VoteCut
19.1
10.1
19.6
19.6
17.8
8.7
9.5
18.9
18.9
DINO s16, DINOv2 s14, DINOv2 b14
VoleCut+ SAM
supervised SA- dataset
21.9
9.1
10.7
19.7
19.7
18.4
9.2
9.9
19.1
19.1
Table 14: Training and inference time of different methods. For fair comparison, all methods are evaluated the same hardware configurations.
Training Time (hours in total)
Inference lffficiency (seconds per image)
Direct
MaskCut (N=3)
MaskCut (N=10)
VoteCut
unMORE.ue 10.1
MaskCut (N=3)
MaskCut (N=
VoteCit
MnMOR
Object Discovery
11.3
33.7
5.1
45.3
Training
UnSAM 90.0
CuLLE 75.0
CUVLER 60.0
unMORE 30.0
UnSAM 3.0
CuILER
CUVLER 0.1
MiMORE
Detectors
0.1
0.1
19

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundanl Reasoning
Multi-Object Reasoning Overview
Center Reasoning
Boundary
Reasoning
Split Proposals
Iteration
Iteration
Iteration 4
Center Reasoning Details
Proposal Splitting
Boundary Reasoning Details
Proposal #1
Proposal #2
Proposal
Proposal
Proposal #5
Proposal #6
Iteration
Iteration 2
Iteration
Figure 10: Multi-object reasoning with object center and boundary representations multi-object image.
Input
GT Mask unMOREatee
Input
GT Mask
unMORF
Input
GT Mask
unMOREaise
Figure 11: Failure cases unMORE
20

---

unMORE: Unsupervised Multi-Object Segmentation via Center- Center-Boundary Reasoning
A.16. Details of COCO* Validation Set
In COCO*, we exhaustively label objects in the COCO val2017 dataset, which comprises 5,000 images and originally contains 36,781 instances across 90 categories. We have added 197 new object categories and labeled previously unannotated objects within the original COCO categories. In total, COCO* includes 5,000 images, 287 categories, and 47.117 labeled objects. Details for the annotated categories are provided in Table 15. We use SAM (Kirillov et al., 2023) to expedite the labeling process. We label each object of interest with tightest bounding box around it. This bounding box, along with the full image, is then fed into the SAM model to generate a dense binary mask.
21

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundatt Reasoning
Table 15: Details of COCO* validation set. This table includes the unique class IDs, class names and the number of newly labeled objects that belong to each class. Specifically, the newly introduced classes are assigned with IDs from 100 to 297. Apart from the 197 new categories, we also label objects belonging to the original COCO classes (the id between 1-90) that not labeled in COCO validation 2017. In summary, we have labeled 10,336 objects in addition to the original 36,781 objects on COCO validation 2017, resulting in 47,117 objects on 5,000 images.
id class name count id class name count
id class name
count
id class name
count
3
car
128 tissue
184
183
cabbage
24
247 corn
9
11
fire hydrant
129 rice
27
184
cucumber
39
248 plum
5
15
bench
6
130 painting
445
185 calendar
13
249 MP3 player
6
17 cat
131 board
40
186 pinapple
19
250 garlic
3
20 sheep
132 ballon
49
187 key
11
25/ scallion
33
suitcase
133 camera
71
188 pumpkin
6
252 noodle
9
44 bottle
175
134 handler
73
189 ball
15
253 soup
14
47 cup
44
135 soap
19
190 calculator
254 onion
6
49 knife
136 brush
37
191 flashlight
8
255 sausage
20
50 spoon
137 shower
21
192 ush
13
255 vegatable
19
5/ bowl
17
138 bectroot
193 potato
15
257 fishbowl
4
53 apple
19
139 meat
102
194 ipad
5
258 wallet
3
56 broccoli
I40 bridge
11
195 pad
40
259 buoy
15
57 carrot
11
141 grape
55
196 banner
174
260 roadblock
56
59 pizza
142 cheese
10
197 funnel
3
26/ chocolate
12
61 cake
12
143 clothes
102
198 blender
30
262 shell
7
62 chair
34
144 box
186
199 name tag
125
263 wool
5
63 couch
145 curtain
228
200 jar
74
264 avocado
67
dining table
146 beans
15
201 flag
156
265 charger
9
70 toilet
10
147 dustbin
131
202 peach
4
266 card
4
75 remote
148 broom
203 radio
267 coin
4
76 keyboard
63
149 stand
86
204 helmet
466
268 wire
9
77 cell phone
4
150 statue
69
205 cart
32
269 piano
6
79 oven
11
151
fries
16
206 toothpaste
14
270 chinaware
13
81 sink
35
152
plastic bag
104
207 coconut
271 balance
82 refrigerator
153 blanket
71
208 salmon
21
272 pancake
3
84 book
18
154
bathtub
38
209 tongs
1
273 pepper
8
86 vase
16
155 stationary
59
210 CD player
34
274 eggplant
2
101 cabinet
291
156 sauce
47
211 heater
18
275 napkin
18
102 carpet
65
157 poster
194
212 air conditioner
12
276 table stand
3
103 lamp
495
158 sail
213 butterfly
22
277 kiwifruit
104 basket
87
159 rhino
214 tent
15
278 fig
1
105 pillow
312
160 paper
142
215 salad
18
279 soother
2
106 mirror
67
161 hook
28
216 spagatti
280 pomelo
2
107 pot
227
162 hand dryer
217 gravestone
28/ guita
108 hat
179
163
tomato
53
218 arcade game machine
282 screen
15
109 scarf
13
164
lemon
18
219 chips
12
283 callbox
2
110 flower
253
165 snail
220 fish
16
284 map
111 applicance
82
166
candle
70
22/ pig
285 coffee machine
112 can
71
167 teapot
46
222 dish
71
286 dishwasher
113 skate shoe
189
168
moon
223 CD
30
287 soap stand
114 glove
143
169 strawberry
26
224 doll
29
288 shelf'
12
115 stove
45
170 paperbag
20
225 watermelon
6
289 prize
0
116 watch
38
171 lid
30
226 cherry
4
290 tower
5
117 ornament
187
172 earphone
32
227 cream
12
29/ picture
13
118 oar
173 egg
28
228 toy
43
292 vent
5
119 speaker
90
174 butter
10
229 pomegranate
293 baggage tag
32
120 printer
22
175 tap
220
230 rolling pin
294 biscuit
121 monitor
176 fan
38
231 envolop
3
295 telescope
1
122 basin
75
177 switch
128
24/ sticker
51
296 pear
5
123 road sign
555
178 telephone
34
242 dough
7
297 ferris wheel
2
124 towel
213
179
socket
114
243 pan
12
298
lizard
125 ashtray
7
180
bag
86
244 peanut
126 plate
190
181
quilt
46
245
billboard
154
127
bread
87
182
tank
11
246
ladder
22

---

unMORE: Unsupervised Multi-Object Segmentation via Center- Center-Boundary Reasoning
A.17. More Results and Analysis of Object-centric Representations
In this section, we provide more insights into the comparison between our proposed center-boundary representations and self-supervised features. In particular, experiment with four pre-trained models from DINO and two pre-trained models from DINOv2, with different patch sizes and/or model parameter scales.
Motivated by NCut (Shi & Malik, 2000) algorithm, given a set of image features, we construct a weighted graph. The weight on each edge is computed as the similarity between features, formulating an affinity matrix W. Then, we solve an cigenvalue system (D a = ADx for a set cigenvectors z and eigenvalues X, where D is the diagonal matrix. In Figures & 13 14 & 15 & 16, we visualize the eigenvectors corresponding the 2nd, 3rd, and 4th smallest eigenvalues. Specifically, we resize all eigenvectors to be the same size the source image.
In practice, methods like TokenCut (Wang et al., 2023b) and CuVLER (Arica al., 2024) directly use the eigenvector corresponding to the 2nd smallest eigenvalue and perform clustering onto it.
From Figures 12 & 13 & 14 & 15 & 16, we observe that segmenting objects via grouping pre-trained self-supervised features: 1) focuses on large objects that dominate the image, while ignoring objects with smaller sizes, 2) tends to capture Boikgrouuddforegrouuu
contrast, instead of objectness. For example, in Figure 12, only object with large size can be discovered by clustering eigenvectors. In Figure 13, the two "keyboards", two "monitors", and two "speakers" are hard to be distinguished into separate clusters. Such behaviors fundamentally due to the training of self-supervised features only involving image-level contrast, which can hardly lead to fine-grained object understanding.
semantic similarity
"bed"
In contrast, as shown in the last row of Figures 12 & 13 & 14 & 15 & 16, our proposed center and boundary representation captures more fine-grate fine-grained properties that directly reflect objectness, which naturally leads to better object discovery results. should be noted that the merged center field and merged boundary distance field are derived by combining all proposals with their predicted center field and boundary distance field, instead of predicted in one pass.
Object
Discovery Result
unMORE (Ours)
Image
MaskCut
eigenveecto smallest eigenvaliee
eigenvector
eigenvector
for smallest eigenvalue
eigenvector 3 smallest eigenvalue
eigenvector for smallest eigenvalue
3dd smallest eigenvalue
4 smallest eigenvaliae
DINO_ste s16
DINO b16
DINO_s
DINO
DINOv2_shP 814
DINOv2 b14
Representations
Merged
Merged Field
Houndary
Distance Field
Figure 12: Comparison between DINO/DINOv2 features with proposed boundary-center representations. The eigenvectors are reshaped to be the size of the image. The last row shows the illustrations for the proposed center and boundary distance representations (predicted).
23

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundang Reasoning
Object Discovery Result
enMORE dic (Ours)
Image
MaskCut
VoteCut eigenvector for smallest cigenvaluee
cigenvecior for 24 smallest eigenvalue
cigenvector For smallestt eigenvalue
eigenvector for 4 smallest eigenvalue
eigenvector for smallest cigenvalue
eigenvecior for
4" smallest eigenvalue
DINO_st ,16
DINO_s8 $8
DINO bR
DINOV2
Represcutations
Merged
Merged Center Field
Boundary
Distance
Figure 13: Comparison between DINO/DINOv2 features with proposed boundary-center representations. The eigenvectors are reshaped to be the size of the image. The last row shows the illustrations for the proposed center and boundary distance representations (predicted).
Object Discovery Result
un MORR (Ours)
Image
MuskCut
cigcnvedtor For 2d smallest eigenvalue
eigenvector for 3d smallest eigenvalue
eigenvector for 4 smallest eigenvalue
eigenvector for 2dd smallestt cigenvaluc
eigenvector 3ed smalksst eigenvaluc
eigenvector
4 smallest eiganvaluc
DINO s16
DINO_s8
DINO b8
D110~2,514
DINOv2 bl4
Our Representations
Menged
Merged
Boundary
Distunce Field
Figure 14: Comparison between DINO/DINOv2 features with proposed boundary-centee representations. The eigenvectors are reshaped be the size of the image. The last row shows the illustrations for the proposed center and boundary distance representations (predicted).
24

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundanl Reasoning
Object Discovery
Result
unMORE (Ours)
Image
VoteCat
eigenvectir
eigenveztor 3 smallest eigenvalue
oigenvector For 40 smallest cigenvalue
eigenvector
for 3 smalllest eigenvalue
eigenvecter
2 smallest eigenvalue
2 smallest cigenvalue
smallest eigenvalue
DINO_s16
DINO b16
DINOv2_s14
Our Representations
Merged
Merged Cente Field
Boundary
Distance Field
Figure 15: Comparison between DINO/DINOv2 features with proposed boundary representations. The eigenvectors are reshaped to be the size of the image. The last row shows the illustrations for the proposed center and boundary distance representations (predicted).
Object Discovery Result
unMORE
Image
MackCut
VoteCut eigenvector for 2dd smallest cigenvaluc
sigenvector 2 smallest eigenvalue
cigenvecior 3d smallesst eigenvalue
eigenvector
eigenvector for cigenvaluee
eigenvector for smallest cigenvalue
4 smallost eigenvaluc
DINO s16
DINO b16
DINO_s8
DINOv2_s14
b14
Our Representations
Merged
Boundary
Merged Center Field
Distance Field
Figure 16: Comparison between DINO/DINOv2 features with proposed boundary-center representations. The eigenvectors are reshaped to be the size of the image. The last row shows the illustrations for the proposed center and boundary distance representations (predicted).
25

---

unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning
A.18. Efficiency of Direct Object Discovery
For our method of direct object discovery the COCO* validation as described in Group of Sec 4.1, in implementation, the maximum number of iterations optimize proposal is set to be 50. Nevertheless, in practice, shown in Figure 17 which illustrates the relationship between the average number of pixels to increase or decrease and the number of optimization steps, we observe that all proposals tend to converge after just 10 iterations.
22.5
propusal update (pixels)
15.0
10.0
7.5
10
20
30
40
50
step
Figure 17: The relationship between the average number of pixels to increase/decrease and the number of optimization steps.
26


Reviews:
[{"review_id": "PG7YLN3U8Z", "reviewer": "Reviewer_QK7x", "summary": "The paper proposes OCN, which improves unsupervised multi-object discovery by introducing three objectness scores to measure existence, centers, and boundaries, along with a reasoning module to distinguish objects. The model is trained by bootstrapping rough masks from DINOv2 and refined through distillation with inductive biases, leading to superior performance over CutLER and CuVLER.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address all concerns below. An anonymous PDF with figures and tables is available at: <https://github.com/icml5450/icml5450/blob/main/FiguresTables.pdf>\n# Q1: Title and terminology\nA1: Thanks for this advice. We will consider an alternative title \"Unsupervised Multi-Object Segmentation via Center-Boundary Aware Reasoning\". In addition, we will ensure that the relevant terminology throughout the paper is updated accordingly.\n# Q2: Table and Figure based on object count\nA2: We present a detailed evaluation on COCO* validation dataset based on object count in the attached ***Table 1*** with two more metrics AR$^{box}$/AR$^{mask}$ as requested by reviewer **CHsG**. We can see that, when the number of objects in each image is rather small (e.g., [0 - 4]), the results of top-performing baselines VoteCut/CuVLER are comparable to our method, all yielding high scores. However, as the number of objects per image increases (e.g., $\\geq5$ objects), our OCN$_{disc}$/ OCN consistently outperforms all baselines by growing margins, demonstrating the superiority of our method in dealing with challenging crowded images. \n\nNotably, UnSAM achieves high AR$^{box}$/AR$^{mask}$ scores (used in the original UnSAM paper to measure the average recall rate without limiting the number of predictions), but its AR$^{box}\\_{100}$/AR$^{mask}\\_{100}$ scores (only considers the top 100 predictions per image and commonly adopted for object segmentation) are clearly lower. This is because UnSAM focuses on excessively partitioning images by clustering granular segments, which sacrifices the accuracy of object discovery, but tends to oversegment objects. This is also qualitatively validated in attached ***Figure 5*** and ***Figure 6***.\n\nWe present more qualitative results in the attached  ***Figure 1*** for multi-object reasoning.\n# Q3: Refinement through objectness\nA3: This is an insightful point. We compare VoteCut and our OCN$\\_{disc}$ on COCO train2017 and ImageNet val splits. As shown in the attached ***Table 2***, our OCN$\\_{disc}$ is on par with VoteCut on ImageNet val, validating that our OCN$\\_{disc}$ indeed learns valid objectness from rough masks generated by VoteCut on the train split of ImageNet. Since most images of ImageNet have a single object, it is expected that our OCN$\\_{disc}$ performs similarly to the pseudo label generator VoteCut. However, on the challenging COCO train2017, our OCN$\\_{disc}$ clearly outperforms VoteCut, validating that the learned (refined) objectness by our OCN$\\_{disc}$ can better deal with undersegmentation on multi-object images, whereas VoteCut cannot. \n\nAs shown in the attached ***Figure 2***, rough masks from VoteCut  on both COCO train2017 and ImageNet val are prone to undersegmentation, while OCN$\\_{disc}$ shows a stronger ability to distinguish multiple objects. \n# Q4: Training and inference time\nA4: We report the training and inference time in the attached ***Table 3***. Our OCN$\\_{disc}$ takes 10 hours to train the objectness network and is slower for Direct Object Discovery. However, our subsequent detector OCN requires only 30 hours to train, benefiting from the high-quality pseudo labels from OCN$_{disc}$, while baseline detectors take over 60 hours. Ultimately, the inference speed of our OCN matches that of CutLER and CuVLER.\n# Q5: Add unSAM to Table 2 of the main paper\nA5: We report zero-shot results of UnSAM in the attached ***Table 4*** (will replace Table 2 in the main paper), with two more metrics AR$^{box}$/AR$^{mask}$ as requested by reviewer **CHsG**. We can see that UnSAM achieves the highest AR$^{box}$ or AR$^{mask}$ scores on all datasets, but its other important metrics are rather low. This is because UnSAM tends to oversegment objects, as also confirmed in the attached ***Table 1***.\n# Q6: unSAM in Table 1 of the main paper and its reproduction\nA6: To clarify, all results of UnSAM in our paper are based on its official checkpoints and code. We will rephrase sentences. \n\nIn the attached ***Table 5*** (will replace Table 1 in the main paper), we add three more metrics: AR$^{box}$, AR$^{mask}$, and \"\\# of pred obj.\". Again, we can see that UnSAM achieves very high AR$^{box}$/AR$^{mask}$ scores, primarily because it tends to predict an excessive number of objects. This clearly explains its rather low scores on all other critical metrics commonly-used for object segmentation. \n# Q7: Figure 1 improvement\nA7: We present an updated version in the attached ***Figure 3*** which will replace Figure 1 of the main paper.\n# Q8: Table 3 improvement\nA8: We present an updated version in the attached ***Table 6*** which will replace Table 3 of the main paper.  \n# Q9:  Visualizations in Figures 11–17\nA9: We present new visualizations in the attached ***Figure 4/5/6*** by re-organizing existing materials and adding results of UnSAM on both COCO* validation and zero-shot datasets. We will include them in the main paper for better illustration.", "claims_and_evidence": "The paper does not focus on \"object-centric representation\" as it only learns object segments, unlike SlotAttention, which learns embeddings for each segment. The title and terminology should be corrected to reflect its focus on \"unsupervised multi-object segmentation.\"\n\nThe main contribution is improving unsupervised segmentation in scenes with many objects, but this is not thoroughly analyzed:\n- The tables present only average performance per dataset. Showing performance based on object count would be more informative. While the paper uses COCO*, a multi-object extension of COCO, the analysis is insufficient.\n- The figures display only object centers and boundaries for a single object, including those in Appendix A.13. More qualitative results demonstrating objectness in multi-object segmentation would be beneficial.", "methods_and_evaluation_criteria": "1. The paper aims for fine-grained object discrimination through objectness measurement, but the results are unconvincing due to the lack of performance analysis by object count and qualitative results on multi-object images, as mentioned above.\n2. The model relies on rough masks for supervision, which may introduce errors if they fail to distinguish objects. It would help to show that while the original masks suffer from merging issues, refinement through objectness improves segmentation both qualitatively and quantitatively.\n3. The paper introduces multiple complex modules beyond prior work. It should compare training and inference time, not just list trainable modules in Table 1.\n4. While CutLER and CuVLER suffer from undersegmentation, UnSAM does not. Why does Table 2 omit a comparison with UnSAM?\n5. Why is UnSAM’s performance in Table 1 so low? Can the authors justify this? Also, why reimplement its results instead of comparing directly with the benchmarks reported in the UnSAM paper?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Mentioned above.", "relation_to_broader_scientific_literature": "Object discrimination is a fundamental problem in computer vision with applications across various visual tasks, including scientific problems like cell segmentation.", "essential_references_not_discussed": "Well-cited, as far as I know.", "other_strengths_and_weaknesses": "I appreciate that the authors have updated the paper rather than simply resubmitting a rejected version, addressing prior reviews. Here are my thoughts on additional strengths and weaknesses after reading A.17.\n\nTechnical contribution (strength):\n- While concerns about technical novelty are valid, the paper makes a reasonable contribution by demonstrating how combining objectness components improves multi-object discovery benchmarks.\n- Prior work is well surveyed and properly discussed in Sec 3.2.\n\nPresentation (weakness):\n- The presentation could be further refined for clarity and readability.\n- Figure 1 lacks clarity and should convey a clear message without relying on the text.\n- The ablation study in Table 3 is difficult to read. Instead of describing variants 1–8 in the text, clarify them directly in the table using checkmarks or dashes.\n- Visualizations in Figures 11–17 are strong. Consider curating some for the main paper to provide additional insights beyond numerical results. Adding more baselines, especially UnSAM, would further illustrate how OCN outperforms it qualitatively.", "other_comments_or_suggestions": "Mentioned above.", "questions_for_authors": "Questions 1-5 in \"Methods And Evaluation Criteria.\"\n\nVisual comparison between OCN and UnSAM (extension of Figures 11–17).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7WyEdhBrTg", "reviewer": "Reviewer_rRHV", "summary": "This paper presents OCN, a new two-stage framework for unsupervised multi-object segmentation in images. The proposed pipeline consists of two stages: the first stage involves learning three levels of object-centric representations—object existence, object center field, and object boundary distance field. In the second stage, a center-boundary aware reasoning algorithm is introduced to iteratively discover multiple objects in single images without relying on neural networks or human annotations. OCN demonstrates superior performance compared to existing unsupervised methods across six benchmark datasets, including COCO, achieving state-of-the-art results in object segmentation, especially in crowded scenes where other methods struggle.\n\n##update after rebuttal: Thanks for the rebuttal. My concern has been addressed mostly. I recommend accept this paper. Great work!", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address all concerns below. An anonymous PDF with figures and tables is available at: <https://github.com/icml5450/icml5450/blob/main/FiguresTables.pdf>\n# Q1: Applied on videos\nA1: We agree with the reviewer and conduct the following experiments on YouTubeVIS-2021 dataset to verify the effectiveness of OCN for unsupervised video object segmentation. \n- Dataset: The YouTubeVIS-2021 dataset consists of 2,985 training videos and 421 val videos whose labels are held for competitions. Thus, we split original training videos into two subsets: YouTubeVIS-2021 Train\\# (2,795 videos) and YouTubeVIS-2021 Val\\# (200 videos). \n- Baselines: 1) We compare with CutLER's extension to video domain: VideoCutLER(CVPR24). 2) We also adapt CuVLER to video domain: VideoCuVLER. 3) We adapt OCN to video domain: VideoOCN. \n- Experiments: We follow VideoCutLER: Step-1: generate pseudo labels for unlabeled images. Step-2: generate synthetic videos with images and pseudo labels from Step-1. Step-3: train a Mask2Former model for video segmentation on synthetic videos and labels from Step-2.\n- Results: As shown in the attached ***Table 7***, the baselines VideoCutLER and VideoCuVLER are trained on 2 types of training sets, ImageNet and YouTubeVIS-2021 Train\\#. VideoOCN consistently outperforms all baselines on most metrics without extensively tuning hyperparameters due to limited time. Qualitative results for video segmentation can be found in the attached ***Figure 7***. \n# Q2: Pretrained features and generalization\nA2: This is an insightful point. Like almost all pretrained models, the learned features are always depending on training datasets, and can hardly generalize to extremely different domains due to the fundamental data-driven learning principle. As shown in the attached ***Table 4***, our OCN (trained on natural images) demonstrates excellent zero-shot performance on unseen datasets with diverse types of natural images. Nevertheless, for datasets with significant domain gaps (e.g., medical images), our learned object priors from natural images may not achieve comparable results as expected. To enhance generalization, one potential solution could be to increase the diversity of training datasets. However, this is a non-trivial task and is left for future exploration.\n# Q3: Failure cases and limitations\nA3: We present failure cases in the attached ***Figure 8*** and discuss limitations as follows. \n\n1. The Direct Object Discovery of our OCN$\\_{disc}$ takes time. It could be possible to leverage reinforcement learning techniques to learn an efficient policy network to discover objects.\n\n2. Our method struggles to separate overlapping objects with similar textures, as shown in the attached ***Figure 8***. Adding additional language priors may help alleviate this issue. \n# Q4: Ablation study on pseudo-masks\nA4: We conduct the following ablation study on four types of pseudo-masks:\n- SelfMask[CVPRW22]: For each image, we employ the strong unsupervised saliency detection model SelfMask to predict a salient region as the pseudo label.\n- MaskCut: For each image, we use the first object discovered by MaskCut as the pseudo label.\n- VoteCut: It's used in our paper. \n- VoteCut+SAM: For each image, a rough mask is generated by VoteCut, and its bounding box is used as a prompt for SAM to predict the final pseudo mask. While this setup yields the best pseudo labels, SAM is a fully supervised model, so this ablation is for reference only.\n\nAs shown in the attached ***Table 8***, our method is amenable to all types of rough masks, though their quality affects OCN$\\_{disc}$ performance. While SAM scores highest, its improvement over VoteCut is not substantial, as it still relies on bounding box prompts from VoteCut. Importantly, our method does not depend on specific pretrained features, enabling the use of enhanced pretrained models in the future.\n# Q5: Time consumption and throughput\nA5: We present the time consumption in the attached ***Table 3***. Our OCN$\\_{disc}$ takes 10 hours to train the objectness network and is slower for Direct Object Discovery. However, our subsequent detector OCN requires only 30 hours to train, benefiting from the high-quality pseudo labels from OCN$_{disc}$, while baseline detectors take over 60 hours. Ultimately, the inference speed of our OCN matches that of CutLER and CuVLER.\n\nRegarding the throughput, for each image on average, the number of initial proposals is 1122.7, whereas the number of predicted objects from OCN$_{disc}$ is 8.9. Most initial proposals have low existence scores and are discarded at the first iteration. The Non-Maximum Suppression (NMS) will also remove redundant proposals.", "claims_and_evidence": "The claim of superior performance compared to existing unsupervised methods is well-supported with quantitative results on 6 benchmark datasets, including COCO. Tables and comparisons are provided.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria in the paper make sense for the problem of unsupervised multi-object segmentation in single images", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The design is robust, and it is particularly important to evaluate whether the method performs exceptionally well in scenarios involving crowd images (i.e., cases with multiple objects), as demonstrated in Table 5 of Appendix A.8.", "relation_to_broader_scientific_literature": "I think the the finding in this paper can be applied in the video domain, too.", "essential_references_not_discussed": "The related work included is sufficient to understand the key contributions.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a detailed explanation of the method's motivation and description. The experiments are comprehensive, including comparisons with state-of-the-art methods across diverse benchmarks and an ablation study.\n\n2. The method is both sound and intuitive. Using three levels of objectness to feed into the object reasoning network makes sense, and leveraging DINO's self-supervised features allows it to perform well in crowded scenarios.\n\nWeaknesses:\n\n1. The three levels of object priors are heavily reliant on the pretrained features. This dependence could introduce biases from the training dataset, potentially limiting the model’s generalization capabilities.", "other_comments_or_suggestions": "The paper could provide more discussion of failure cases and limitations, which would give a more balanced view of the capabilities.", "questions_for_authors": "1. Does the method heavily rely on VoteCut when training the Objectness Network? Is there an ablation study to replace the pseudo-masks generated by other methods?\n\n2. In Appendix A.16, the paper mentions that the average number of iterations in the object reasoning module is typically 10. Could this be considered time-consuming? Additionally, what is the average throughput of this module?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XNTzqmne4m", "reviewer": "Reviewer_su1q", "summary": "This paper proposes a multi-object segmentation approach that first trains objectness networks to identify the existence, object center, and object boundary of individual objects, and then use the trained networks to discover objects on images without further training modules. The paper claims that the approach can discover multi-objects more accurately compared to baselines without having access to image annotations.\n\n###  Most of my concerns are addressed. I have updated my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address all concerns below. An anonymous PDF with figures and tables is available at: <https://github.com/icml5450/icml5450/blob/main/FiguresTables.pdf>\n# Q1: Representations for segmentation and generation\nA1: This is an interesting point. First, this paper indeed mainly tackles unsupervised multi-object segmentation.\n\nSecond, our object-centric representations hold potential for generative tasks. We could train a diffusion model to create object center and boundary fields given various prompts. This approach can be expanded to generate complex multi-object images by first creating individual objects and layouts, allowing for controlled sampling from the learned representations to ensure high-quality boundaries and shapes. However, this is non-trivial and left for future exploration.\n# Q2: Novelty of objectness networks\n\nA2: We will consider an alternative title \"Unsupervised Multi-Object Segmentation via Center-Boundary Aware Reasoning\", highlighting our core contribution to the challenging task of unsupervised multi-object segmentation and our key technique of center-boundary aware reasoning algorithm. \n\nRegarding objectness networks, as extensively discussed in Section 2 of the main paper, relevant works mainly use the concept of center or boundary in isolation to tackle fully-supervised tasks. In contrast, we extend the concepts of object existence, object center field, and object boundary field in a joint manner to tackle the challenging unsupervised task. \n\nOverall, our proposed two-stage pipeline, object-centric representation learning followed by multi-object reasoning, clearly departs from existing unsupervised methods such as slot-based approaches. \n# Q3: Hinges on VoteCut\nA3: To clarify, our object representation is independent of any specific self-supervised features or grouping strategies. We conduct the following ablation study on four types of pseudo-masks:\n- SelfMask[CVPRW22]: For each image, we employ the strong unsupervised saliency detection model SelfMask to predict a salient region as the pseudo label.\n- MaskCut: For each image, we use the first object discovered by MaskCut as the pseudo label.\n- VoteCut: It's used in our paper. \n- VoteCut+SAM: For each image, a rough mask is generated by VoteCut, and its bounding box is used as a prompt for SAM to predict the final pseudo mask. While this setup yields the best pseudo labels, SAM is a fully supervised model, so this ablation is for reference only.\n\nAs shown in the attached ***Table 8***, our method is amenable to all types of rough masks, though their quality affects OCN$\\_{disc}$ performance. While SAM scores highest, its improvement over VoteCut is not substantial, as it still relies on bounding box prompts from VoteCut. Importantly, our method does not depend on specific pretrained features, enabling the use of enhanced pretrained models in the future.\n# Q4: Twin negative sample\nA4: Different unsupervised methods leverage self-supervised features and raw images in various ways. In our approach, generating twin negative samples from rough masks is effortless and can be regarded as data augmentation without additional human annotations. Thus, this operation should not be seen as a weakness.\n# Q5: Fairness to fully unsupervised approaches\nA5: Since there is no official definition of what constitutes an unsupervised approach, the fairness criteria for comparison are derived from the extensive body of existing unsupervised methods, notably including recent works such as CutLER, CuVLER, and UnSAM. These methods share a core principle: avoiding the use of human labels while fully utilizing self-supervised features and the derived pseudo labels in various ways.\n\nOur method adheres strictly to this core principle, relying solely on self-supervised features without introducing any additional human annotations. To clarify, all baselines in our paper utilize pretrained self-supervised features. Therefore, it is evidently fair to compare our method with all the unsupervised baselines listed in our paper. \n# Q6: Scale to a large number of objects\nA6: We present a detailed evaluation on COCO* validation dataset based on object count in the attached ***Table 1***. We can see that, as the number of objects per image increases (e.g., $\\geq5$ objects), our OCN$\\_{disc}$/ OCN consistently outperforms all baselines by growing margins, showing the superiority of our method in dealing with a large number of objects.\n# Q7: Segmentation speed\nA7: Time consumption is detailed in the attached ***Table 3***. Our OCN$\\_{disc}$ takes 10 hours to train the objectness network and is slower for Direct Object Discovery. However, our subsequent detector OCN requires only 30 hours to train, benefiting from the high-quality pseudo labels from OCN$\\_{disc}$, while baseline detectors take over 60 hours. Ultimately, the inference speed of our OCN matches that of CutLER and CuVLER.", "claims_and_evidence": "The claims are mostly supported by evidence.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem.", "theoretical_claims": "No theoretical claims are provided.", "experimental_designs_or_analyses": "Experiments are properly designed to support claims.", "relation_to_broader_scientific_literature": "The object-centric representations discussed in the paper is different from that studied in slot-based methods. The object centric representations learned by slot-based methods cannot only be used for segmentation, but also for generalized composition or generation [1][2][3]. The object centric representations in the proposed paper is mainly for object discovery and segmentation.\n\n\n[1] Jiang, J. and Ahn, S., 2020. Generative neurosymbolic machines. Advances in Neural Information Processing Systems, 33, pp.12572-12582.\n\n[2] Wang, Y., Liu, L. and Dauwels, J., 2023, July. Slot-vae: Object-centric scene generation with slot attention. In International Conference on Machine Learning (pp. 36020-36035). PMLR.\n\n[3] Wu, Y.F., Lee, M. and Ahn, S., 2024. Neural language of thought models. arXiv preprint arXiv:2402.01203.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strength**\n\n1. The writing is smooth and easy to follow.\n\n2. The paper proposes a pipeline to deal with multi-object segmentation without explicit human annotations.\n\n3. Extensive experiments are conducted to demonstrate the effectiveness of the approach.\n\n**Weekness**\n\n1. The novelty seems somewhat limited considering that the objectness networks have been widely studied in the literature. The proposed approach simply use them for object discovery.\n\n2. The no supervision claim is somewhat overstated. Though the paper emphasizes it is a fully unsupervised approach and does not require human labels, the approach dose involve label obtaining implicitly. First, the use of pretrained models makes the approach less unsupervised and the accuracy of the object representation hinges on the performance of VoteCut; Second, the approach **needs to create a twin negative sample by cropping the largest rectangle on background pixels excluding the tightest object bounding box** as stated in the paper.  Considering these points, is it fair to compare the proposed approach with fully unsupervised approaches?\n\n3. Though the Multi-Object Reasoning Module is training free by directly using the objectness network, it relies on randomly initialization and iterative update of multiple bounding boxes. Does this scale well to images containing a large number of objects? Appendix A.16 studies the iterations of the approach needs. Is there any analysis on segmentation speed?", "other_comments_or_suggestions": "For novelty concern, I would like to hear more from other reviewers, and I am open to change my rating.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sVCUw2dCl", "reviewer": "Reviewer_CHsG", "summary": "The paper proposes a novel framework for unsupervised object segmentations. It proposes a two stage solution by incorporating an objectiveness network that is trained on an object centric dataset (ImageNet) to predict the existence, location, and boundary of each object, and a reasoning module to generate final predictions based on several heuristics. Experiments show that the proposed methods outperforms existing baselines on COCO and SA-1B dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and address all concerns below. An anonymous PDF with figures and tables is available at: <https://github.com/icml5450/icml5450/blob/main/FiguresTables.pdf>\n# Q1: Include UnSAM in Table 2\nA1: We report zero-shot results of UnSAM in the attached ***Table 4*** (will replace Table 2 in the main paper), with two more metrics AR$^{box}$/AR$^{mask}$ as requested. We can see that UnSAM achieves the highest AR$^{box}$ or AR$^{mask}$ scores on all datasets, but its other important metrics are rather low. This is because UnSAM tends to oversegment objects, as also confirmed in the attached ***Table 1***, as well as qualitative results in ***Figure 5*** and ***Figure 6***.\n\n\n# Q2: Add AR$^{box}$ and AR$^{mask}$ in Table 1\nA2: We present the attached ***Table 5*** (will replace Table 1 in the main paper) by adding three more metrics: AR$^{box}$, AR$^{mask}$, and \"\\# of pred obj.\". The AR$^{box}$/AR$^{mask}$ scores are used in the original UnSAM paper to measure the average recall rate without limiting the number of predictions, but the AR$^{box}\\_{100}$/AR$^{mask}\\_{100}$ scores only consider the top 100 predictions per image and are commonly adopted for object segmentation. In addition, AP scores evaluate the ability to discover more objects with fewer trials (i.e., detections), constituting a balanced view between accuracy and recall. In this regard, existing object segmentation works typically focus on AP scores as they are more informative and less biased.\n\nFrom the attached ***Table 5***, we can see that UnSAM achieves very high AR$^{box}$/AR$^{mask}$ scores, primarily because it tends to predict an excessive number of objects by grouping granular image segments. This clearly explains its rather low scores on all other critical metrics commonly-used for object segmentation. This is also qualitatively validated in the attached ***Figure 5*** and ***Figure 6***, where UnSAM tends to generate oversegmented patches.\n\nWe appreciate UnSAM's effort in effectively reducing the gap between unsupervised and supervised methods in terms of \"unlimited detection\", mainly measured by AR scores. Nevertheless, our method targets at comprehensive and accurate object discovery, measured by the balanced evaluation protocols for object segmentation.", "claims_and_evidence": "The paper's main claim was well-supported with empirical performance gains on a wide range  of benchmarks and metrics, suggesting the proposed method is effective", "methods_and_evaluation_criteria": "The author conducted experiments on many common benchmarks such as COCO, SA-1B, LIVIS following the standard practice of the community. The choices of benchmarks are sound.", "theoretical_claims": "There are no proofs or theoretical claims.", "experimental_designs_or_analyses": "The author provide a set of comprehensive ablation", "relation_to_broader_scientific_literature": "This paper establishes a new state-of-the-art in the field of unsupervised image segmentation by identifying the key limitations of existing approaches such as slot-based ones and clustering-based ones. The proposed two stage solution is well-motivated and novel, and is a clear departure from existing methods. I see this work has meaningful impact to the unsupervised segmentation community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "There are small issues with regard to completeness of experiments. For example, the author did not include LVIS results of UnSAM in Table 2, while the UnSAM paper reported such results.", "other_comments_or_suggestions": "I'm curious about how authors chose the selected metrics for reporting. It seems the results are mostly focused on AP, and only include AR_100. This setup largely follows Cutler. However, it appears the authors of UnSAM focused more on AR. I'm wondering what is the performance on AR_mask (not AR_mask_100) on COCO. In particular, Table 1 of UnSAM paper showed that their results (measured by AR_mask)  is only marginally behind the supervised SAM baseline. Since this work offers considerable gain over UnSAM, it may be able to further reduce the gap between supervised and unsupervised methods, when measured by AR. Incorporating these comparisons would further strengthen the contribution of this work.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a two-stage unsupervised framework that learns hierarchical object-centric representations and performs network-free multi-object reasoning to achieve state-of-the-art segmentation on complex real-world images without using labels.

It has received 4 reviews, with 1x accept and 3x weak accepts.  Authors provided extensive rebuttals with additional experimental results on unSAM and videos etc, convincing reviewers on an acceptance consensus.  The network-free reasoning aspect using object existence, object centers, and object boundaries is interesting and seems effective.
Decision: Accept (poster)