SOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Authors: Shengcao Cao
Jiuxiang Gu
Jason Kuen
Hao Tan
Ruiyi Zhang
Handong Zhao
Ani Nenkova
Liangyan Gui
Tong Sun
Yu-Xiong Wang
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learning
open-world learning
segmentation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a self-supervised approach for open-world entity segmentation with hierarchical structures.

Abstract: Open-world entity segmentation, as an emerging computer vision task, aims at segmenting entities in images without being restricted by pre-defined classes, offering impressive generalization capabilities on unseen images and concepts. Despite its promise, existing entity segmentation methods like Segment Anything Model (SAM) rely heavily on costly expert annotators. This work presents Self-supervised Open-world Hierarchical Entity Segmentation (SOHES), a novel approach that eliminates the need for human annotations. SOHES operates in three phases: self-exploration, self-instruction, and self-correction. Given a pre-trained self-supervised representation, we produce abundant high-quality pseudo-labels through visual feature clustering. Then, we train a segmentation model on the pseudo-labels, and rectify the noises in pseudo-labels via a teacher-student mutual-learning procedure. Beyond segmenting entities, SOHES also captures their constituent parts, providing a hierarchical understanding of visual entities. Using raw images as the sole training data, our method achieves unprecedented performance in self-supervised open-world segmentation, marking a significant milestone towards high-quality open-world entity segmentation in the absence of human-annotated masks. Project page: https://SOHES.github.io.

Published as a conference paper at ICLR 2024
SOHES: SELF-SUPERVISED OPEN-WORLD HIERARCHICAL ENTITY SEGMENTATION
Shengeao Cao Jiuxiang Gu² Jason Kuen²
Hao Tan
Ruiyi Zhang²
Handong Zhao² Ani Nenkova² Liang-Yas Yan Gui'
Tong Sun?
Yu-Xiong Wang
University of Illinois Urbana- Chaamaaiin Adobe Research [cao44, lgui, www)@iilinok. .ddu .dd (jigu, kuen, hatan, rui suanh hazhao, nenkova, benkora,.tunneadddee.comm
60.8
211
46.1
45.9
45.8
Mask AR
22870
-3755
-1856
26.0
33.33
29.1
26.2
33.5
27.8
31.1
18.1
22.5
11.8%4
SA-1B
LVIS
EntitySeg
ADE20K
PACO-LVIS
Figure 1: SOHES boosts open- world entity segmentation with self-supervisio on various im- age datasets. Compared to prior state of the art, SOHES significantly reduces the gap between self-supervised methods and the supervised Segment Anything Model (SAM) (Kirillov al., 2023), yet using only 2% unlabeled image data as SAM.
ABSTRACT
Open-world entity segmentation, as an emerging computer vision task, aims at segmenting entities in images without being restricted by pre-defined classes, offering impressive generalization capabilities unseen images and concepts. Despite its promise, existing entity segmentation methods like Segment Any- thing Model (SAM) rely heavily on costly expert annotators. This work presents Self-supervised Open-world Hierarchical Entity Segmentation (SOHES), novel approach that eliminates the need for human annotations. SOHES operates in three phases: self-exploration, self-instruction, and self-correction. Given a pre- trained self-supervised representation, we produce abundant high-quality pseudo- labels through visual feature clustering. Then, we train a segmentation model on the lseudo-labels, and rectify the noises in pseudo-labels via teacher-student mutual-learning procedure. Beyond segmenting entities, SOHES also captures their constituent parts, providing hierarchical understanding of visual entities. Using raw images as the sole training data, our method achieves unprecedented performance self-supervised open-world segmentation, marking significant milestone towards high-quality open-world entity segmentation in the absence of human-amme annotated masks. Project page: https: SOHESS github. 1o.
INTRODUCTION
Open-world entity segmentation (Qi et al., 2022; 2023) is an emerging vision task for localizing se- mantically coherent visual entities without the constraints of pre-defined classes. This task, in con- trast to traditional segmentation (Long et al., 2015; Chen et al., 2017; He et al., 2017; Kirillov et al., 2019), aims at creating segmentation masks for visual entities inclusive both "things" (countable
Work done during internship Adobe Research.

---

Published as a conference paper at ICLR 2024
objects such as persons and cars) and "stuff" (amorphous regions such as and sky) (Kirillov et al., 2019; Qi et al., 2022), without regard for class labels. The inherent inclusivity and class- agnostic nature enable open-world entity segmentation to perform strongly on unfamiliar entities from unseen image domains, a frequent real-world challenge in applications such as image editing and robotics. A prominent model for this task is Segment Anything Model (SAM) (Kirillov al., 2023), which has garnered enthusiastic attention for its impressive performance in open-world seg- mentation. However, the efficacy of models like SAM depends on the avilability of extensively annotated datasets. To illustrate, SAM is trained on SA-1B (Kirillov et al., 2023), vast dataset comprising 11 million images and an enormous amount of 1 billion segmentation masks. While au- tomated segmentation plays a central role in building SA-IB, human expertise and manual labor are similarly important, where it takes 14 to 34 seconds to annotate a mask. Meanwhile, it challenging for human annotators to produce segmentation masks at consistent granularity, because there is no universally agreed definition of objects and parts. This reliance on intricately annotated datasets and considerable human effort raises a compelling question: Can we develop high-quality open- world segmentation model using pure self-suupeviisin? The prospect of learning from unlabeled raw images without the need for expert annotations is highly appealing.
In fact, self-supervised visual representation learning (Chen et al., 2020; He et al., 2020; Caron al., 2021; He et al., 2022b) has already shown promise. Such models can effectively exploit useful training signals from purely unlabeled images, resulting in high-quality visual representations that are comparable with those achieved via supervised learning. However, mainstream self-supervised representation learning approaches typically learn holistic representations for whole images, without distinguishing individual entities nor understanding region-level structures. As a result, they cannot be directly used to achieve open-world entity segmentation. Our key insight to bridge this gap is that an intelligent model can not only learn representations from observations, but can also self-evolve to explore the open world, instruct and generalize itself. continuously refine and correct its predictions in self-supervised manner, and ultimately achieve open-world segmentation.
Following this key insight, we propose Self-supervised Open-world Hierarchical Entity Segmenta- tion (SOHES), a novel approach consisting of three phases - 1) Self-exploration: Starting from pre-trained self-supervised representation DINO (Caron et al., 2021), we generate initial pscudo- labels to learn from. By clustering visual features based on similarity and locality, we can discern semantically coherent continuous regions that likely represent visually meaningful entities. 2) Self- instruction: Our initial pseudo-labels are constrained by the fixed visual representation. To refine the segmentation, we train Mask2Former (Cheng et al., 2022) segmentation model on the ini- tial pseudo-labels. Even though the initial pseudo-labels are noisy, learning a segmentation model on them can "average out" the noises, thus predicting more accurate masks. 3) Self-correction: Building upon these more accurate predictions, we employ a teacher-student mutual-lcarning learning frame- work (Tarvainen Valpola, 2017; Liu et al., 2020) to further reduce the early-stage noises and adapt the model for open-world segmentation. Throughout the three phases, we rely solely on the raw im- ages, without any human annotations. Equally significantly, due to the compositional nature of things and stuff in natural scenes, our model learns not just to segment entities but also their con- stituent parts and finer subparts of these parts. During the self-exploration phase, we generate a hierarchical structure of each visual entity from individual parts to the whole. This hierarchical segmentation approach enriches our understanding of visual elements in an open open-world context, ensuring a more comprehensive and versatile application.
To summarize, our key contributions include:
We propose Self-supervised Open-world Hierarchical Entity Segmentation (SOHES) address the open-world segmentation challenge. We demonstrate the potential of high-quality open-world segmentation by adapting sulf-supervised representations rerfeppririieeeeeeeettttiiooo and learning solely from unlabeled data. We develop a method to generate over 100 segmentation masks per image as high-quality pseudo- labels by clustering self-supervised visual features.
We leam to segment entities and their constituent parts and perform hierarchical association be- tween visual entities. This hierarchical segmentation approach provides a multi-granularity anal- ysis of visual entities in complex scenes.
We achieve new state-of tftte-of-theaane t-ttheeart art performance in self-supervised woen-world segmentation, which enhances mask average recall (AR) on various datasets (e.g., improving AR on SA-1B (Kirillov et al., 2023) from 26.0 to 33.3) and closes the performance gap between self-supervised and su- pervised paradigms, as illustrated in Figure
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Open- world visual recognition. Open-world recognition (Scheirer et al., 2012; Bendale & Boult, 2015; 2016) aims to recognize and classify visual concepts in an evolving environment where the model encounters unfamiliar objects, which challenges traditional models trained to recognize fixed set of classes. The task has been extended from classification to detection (Bansal et al., 2018; Dhamija al., 2020; Joseph al., 2021; Jaiswal al., 2021; Kim al., 2022), segmentation (Hu et al., 2018; Wang et al., 2021; 2022a; Kalluri et al.. 2023), and tracking (Liu et al., 2022). In particular, open-wer open-world world entity segmentation (Qi al., 2022; 2023) segments entities into semantically meaningful regions without regard for class labels. In this work, we further expand the scope to whole entities and their constituent parts.
Self-supervised object localization/discovert Localizing objects from images in self self-suuee manner requires learning the concept of objects from visual data without any human annotations. Early explorations (Vo et al., 2019; 2020; 2021) formulate an optimization problem on graph, where the nodes are object proposals (e.g.. by selective search (Uijlings et al., 2013)) and the edges constructed based on visual similarities. Following the observation that the segmentation of the most prominent object can emerge from DINO (Caron et al., 2021), Siméoni et al. (2021; 2023); Wang et al. (2022b) learn object detectors from saliency-based pseudo-labels. Meanwhile, Wang et al. (2022c; 2023) generate pseudo-labels by extending NormCut (Shi & Malik, 2000), and Cao al. (2023) cluster semantically coherent regions into pseudo-labels. We share common multi- phase learning paradigm with these prior methods, where pseudo-labels are first discovered from self-supervised representations, and then a detectiion/segmentation model learned. However, we contribute novel and improved designs to each phase, including 1) a glloallto-lloee clustering algo- rithm for high-quality pseudo-labeling, 2) a hierarchical relation learning module, and 3) a teacher- student self-correction phase.
APPROACH
In this section, we first provide an overview of Self-supervised Open-world Hierarchical Entity Seg- mentation (SOHES) and then present its three learning phases in the following subsections. Build- ing upon and significantly enhancing the pseudo-label discovery and learning paradigm in prior self-supervised object discovery work (Siméoni et al., 2023; Wang et al., 2023; Cao et al., 2023), SOHES consists of three phases: self-exploration, self-instruction, and self-correction, shown in Figure 2. 1) In Phase self-exploration, we start from a pre-trained self-supervised representation DINO (Caron et al., 2021) with VIT-B/8 (Dosovitskiy et al., 2020) architecture, and initiate exploration on unlabeled raw images. Our strategy is based on agglomerative clustering (Hastic al., 2009), and organizes image patches into semantically consistent regions automatically. 2) With these pseudo-labels, we begin Phase 2 self-instruction. We train a segmentation model com- posed by a DINO pre-trained ViT backbone (Caron et al., 2021), ViT-Adapter (Chen et al., 2022) (for generating multi-scale features from ViT), and Mask2Former (Cheng et al., 2022) (for the fi- nal mask prediction). Through self-instruction, our segmentation model can learn from common visual entities in different images and generalize better than the initial pseudo-labels produced by the frozen ViT backbone. 3) In the final Phase self-correction, we exploit more self-supervision
Phase I: explorxxpoootion Generate initial peeudo-labete
Phase 2: Self-instruction Learn from initial pse labell
Phase 3: celf-corrertion
Improne oser initial pseado-ttaells
Unldaded
Tescler Segmentation Model
Row linges
Soguratttion Model
Initial Paeado-labes
Supervte
Tescher's lubels
Masit Fomeer
Froem DINO Vir
Chuster
Add-ddeper
Strskent Segmentaiiin ium Model
Superviier
Supervier
(Iuitaa--Boucck)
Teurus Branch)
Figure 2: Three phases SOHES. In the first eelf-exploration phase, we cluster visual features from pre-trained DINO to generate initial pseudo-labels unlabeled images. Then in the self- instruction phase, a segmentation model learns from the initial pseudo-labels. Finally, in the self- correction phase, we adopt teacher-student framework further refine the segmentation model.
3

---

Published as a conference paper at ICLR 2024
Step 2: Local Re-chustring
Step 3: Mask Refineemen
Step Global Clustering
Step 4: Hierarehy Analysis
Zeder
Vlrre
Bodan
No valid
Emtrect
Wllrge
internal maok
Tratume
Mina
Figure 3: Self-exploration phase for generating initial pseudo-labels. This phase consists of four steps. We first merge image patches into regions with high visual feature similarities, then zoom in on the small candidate regions and re-cluster the local images to better discover small entities. After that, we refine the mask details and identify the hierarchical structure among the masks.
signals to lift the limit induced by noises in the initial pseudo-labels. Inspired by semi- remi-supervised learning (Tarvainen Valpola, 2017; Liu et al., 2020), we employ a teacher-student mutual-lcarning framework, allowing the student to learn from the improved pseudo-labels generated by the teacher.
3.1 SELF-EXPLORATION: GENERATE INITIAL PSEUDO-LABELS
In the self-exploration phase, we generate initial pseudo-labels with several steps delicately designed to include potential entities and their constituent parts of diverse categories. We take a global- to-local perspective to first create candidate regions at the global level, and then investigate local images to accurately discover small entities. In particular, we begin by clustering patch-level self- supervised features to generate pool of candidate regions, then filter and refine such candidates into initial pseudo-labeled masks, and finally analyze the hicrarchical structure among them. Figure depicts this process with visual examples.
Step is a global clustering procedure, which merges image patches into semantically meaningful regions. Given unlabeled image with resolution S x S, use DINO ViT-B/8 to extract its visual features (fi...., f
corresponding to each 8 x patch. Then, we merge these patches in bottom-up, iterative manner. The initial seed regions are exactly these x 8 patches. In each iteration, we find the pair of adjacent regions (i,j) with the highest cosine feature similarity (fi (Il(ll [lf, (2). These two regions i and j are merged into new region k. The visual feature of the merged region computed as fx f.fe+aaf1, where a, aj are the areas of the regions i, j. a,ta, After replacing regions and j with the new merged region k, we continue with the next iteration.
mmagge, as criterion for stopping the merg- ing procedure. In general, the highest cosine feature similarity (among all unmerged region pairs) decreases as more regions threshold Omeegge.t (t E {1, {1...., m)). ). we record the merging results that have been obtained so far. Consequently, we can generate sets of regions, covering various granularity levels. We mix these sets into a pool of regions that may overlap with each other. Non-maximal suppression (NMS) is applied to remove duplicate regions. The thresholds m Im can be determined based on the
We set series of merging thresholds m
merged. When the highest cosine feature similarity goes below one
desired number of pseudo-labed per image (see Appendix D).
Step local re-clustering. In the first step. we have generated large pool of image regions that may correspond to valid visual entities. However, many small regions tend to be noisy and lack meaningful content. We adopt a global-to-local perspective to re-examine the regions that are smaller than 0mmai % of the total image area. For each small candidate region, we crop a local image around it, resize it to x S', and re-cluster it with the same procedure as in Step to obtain subregions of the local crop. Subregions that intersect with the boundaries of the crop are discarded, because they are incomplete within the local crop context. The remaining subregions, along with regions larger than &mmll % of the whole image (from Step 1), form initial pseudo-labels. By "zooming in" on the small candidate regions and repeating the clustering procedure at a finer scale, can better remove noisy pseudo-labels and improve the quality of the remaining ones.
4

---

Published as a conference paper at ICLR 2024
Whole
5
Q
XW2
Query
0
1
0
Query
Part
Maok 2
Xhrok
Query
xW;
0
Query
0
0
0
Query
1
1
Subport
Mark
Mucic
Mack d
Query
0
0
0
Figure 4: Ancestor relation prediction in the self-instruction phase. The prediction target, a binary matrix of ancestor relations, is constructed from the hierarchical structure identified in the self-exploration phase. The ancestor prediction head uses two linear mappings Wi, W2 to transform the query features and learns to predict the target ancestors.
In Step 3, we leverage the off-the-shelf mask refinement model CascadePSP (Cheng et al., 2020) to further refine the boundaries of the pseduo-labe masks. We compute the mask IoUs (intersection- over-union) between the pseudo-labels before and after undergoing the refinement step, and remove the ones that have poor IoUs because they are likely noisy samples.
Finally, Step focuses on identifying the hierarchical structure embedded within the set of pseudo- labels, which is represented as forest structure (ie., set of trees) where the roots are whole entities, and their descendants are parts and subparts, etc. We test each pair of pseudo-labels and j to determine their hierarchical relation: If 1) over Ocovrr pixels of pseudo-label are also in pseudo- label (meaning that is covered by j), and 2) less than Ocover % pixels of pseudo-labe j are in pseudo-label i (meaning that j larger than i), then pseudo-label ji is an ancestor of in the hierarchy forest. The smallest ancestor of i is the direct parent of i. By testing the pixel coverage between pseudo-labels,
figure out the hierarchical structure of pseudo-labels.
3.2 SELF-INSTRUCTION: LEARN FROM INITIAL PSEUDO-LABELS
1) The initial pseudo-labels from the previous self-exploration phase contain noises. How leverage self-supervised learning signals to "average out" the noises? 2) Existing general-purpose segmentation heads cannot predict the hierarchical relations among masks. How to learn the hierarchy forest from the previous phase?
In the iefffinstruction phase, we need to address two problems:
To address the first problem, we train segmentation model to learn and generalize from the ini- tial pseudo-labels. Through this procedure, the segmentation model can observe valid entities from pseudo-labels which are more frequent than noises, and thus accurately segments unseen images. The model is composed of a ViT-based backbone and a Mask2Former (Cheng et al., 2022) seg- mentation model. In particular, the backbone is constructed by the same DINO (Caron et al., 2021) pre-trained ViT, and ViT-Adapter (Chen et al., 2022) for producing multi-scale visual feature maps. The VIT backbone is not fixed, and thus we can adapt its features for the segmentation task.
To accomplish the hierarchical segmentation task, we attach a novel ancestor prediction head to Mask2Former, which predicts the hierarchical relations among the predicted masks. In parallel to the existing mask and class prediction heads, our ancestor prediction head operates on the query features Q R NxC, where N is the number of queries and C' is the query feature dimension. As shown in Figure 4, the learning target of the ancestor prediction is a non-symmetric binary matrix representing the ancestor relations P (0, N where Pt, ancestor of mask j, and P.ù = O otherwise. is worth noting that a mask may have no ancestors (as root in the hierarchy forest), if mask i whole entity; mask i may also have more than one ancestor (as a deep node in the forest), if mask i is a part of another part. The ancestor prediction is formulated as:
represents that mask is an
P = sigmoid (QWW))(W))) /VC) E NRN
where Wi, W2 CCC are learnable weights for two linear transformations. We use two different linear mappings since the ancestor relations are asymmetric. They are optimized via a binary cross- entropy (BCE) loss Lancestor BCE(P. P). At inference time, to reconstruct the forest structure from the binary ancestor relation predictions. Different from prior transformer-basee hierarchical segmentation methods like Group ViT (Xu al,, 2022) which
employ topological sorting
5

---

Published as a conference paper at ICLR 2024
Teacher Branch
Wesk'y Aggumnidd Imagee (wea(ls)
Tocche's
Bacchee
Uniuledal Imuge ly
Teacher (G, Oscocher
Ireacher
Strougly Angmeretd Image Temoaplle)
Studkut's Predictins (ing@1eaee
EMA
Unhibeled Image I
Student M (10 eeuuees
Optimioc
Treral
Strougly Asgmentted Image Tetoog(h)
Studeat's Prodictions
Selforpiorn
Inätial
Finitial
Pscudo labels
Vinna
Initial Branch
Figure 5: Teacher-student mutual-learning in the self-correction phase. We initialize both the teacher and student with the segmentation model learned in the self-instruction phase, which pro- duces better segmentation predictions than the initial pseudo-labels. The student receives supervi- sion from the teacher's 's pseudo-labels and the initial pseudo-labels. The teacher is updated as the exponential moving average (EMA) of the student.
constrained by the pre-defined number of hierarchical levels, our method is able to predict a variable number of levels and entities.
3.3
SELF-CORRECTION IMPROVE OVER INITIAL PSEUDO-LABELS
Although we have elaborately built a pseudo-labeling process in the self-exploration phase, it is still based on fixed self-supervised visual representation that is not optimized for image segmentation. Consequently, there may still be initial pseudo-labels that are noisy and can negatively affect our model. Meanwhile, we observe that the segmentation model learned through the self self-instruction phase can predict masks that are more reliable and accurate than the clustering results from the self-exploration phase. Motivated by this observation, in the final self-correction phase, we further bootstrap our model by learning from itself and mitigating the impact of noises in the initial pseudo- labels. To achieve self-conrection, we adopt a supi-supee approach that is based teacher- student mutual- -eaunnin (Tarvainen & Valpola, 2017; Liu et al., 2020).
The self-correction phase starts off by initializing two separate segmentation models which are exact clones of the segmentation model produced by the self-instruction phase. We denote one segmenta- tion model the student model MM. suudem), which is actively updated through gradient descent; the other segmentation model is the teacher model .M1, Oreacher), which is updated every iteration as an exponential moving average (EMA) of the student: Oreacher +- mm + (1 (11-m]p mm) student, where m E (0, 1) is the momentum. The student receives supervision from both the initial pseudo-labels and the teacher's pseudo-labels. Thus, the total loss computed as:
Laotal = Tuwy((M (Tanonn (T1), Oxtneem), ), Yimitaa) + Ireg(M (M (Tarong(T)) Osusdem) ). Yiescher):
(2)
where 1 and 12 are two image batches, Yiiital is the initial pseudo-labels on 4, Yuscher is the teacher's pseudo-labels by thresholding the predictions M(Tweak (I2), ), Tsuong and Tweak denote strong and weak data augmentations respectively, and Lueg is the segmentation loss which is composed of a classification loss Lek. a mask prediction loss Lmmast and our ancestor prediction loss Lancestor- Figure illustrates the steps in our teacher-student learning approach.
To obtain more reliable supervision from the teacher's predictions (Twaka (T2), Queachr)) we keep only those masks with confidence scores exceeding Oscore to form the pseudo-labels Yreacher. We observe that the teacher model tends to be less confident when segmenting smaller entities. If the threshold Ascore is fixed across all masks, it would result in too few pseudo-labels with small areas, and consequently, the student's 's small entity segmentation performance and overall performance would be impaired. Therefore, we leverage dynamic threshold:
Sscoro (1 (1 a) (0soorc, large O small) + @scorc. small
(3)
where E 1) represents the area ratio of the predicted mask the whole image, 7 1 is and O small < fucore, large are the pre-defined thresholds for the smallest and largest mask, respectively. This dynamic threshold across different scales allows us better balance small, medium, and large entities in the teacher's pseudo-labels, and encourages the student model to segment small entities more accurately.
a

---

Published as a conference paper at ICLR 2024
Table 1: Zero-shot evaluation on various image datasets. SOHES sets new state-of-the-art self- supervised open-world entity segmentation performance. The collection of the evaluation datasets represents diverse classes in an open world and includes both whole entities and parts, Mean- while, using just 2% unlabeled images SAM, SOHES significantly closes the gap between self- supervised methods and the supervised SAM. The evaluation metric is average recall (AR).
Datasets w/ Whole Entities
Datasets w/ Parts
Supervision
Method
COCO
LVIS
ADE
Entity
SA-IB
PuN
PACO
Supervised
SAM (Kirillov et al., 2023)
49.6
46.1
45.8
45.9
60.8
28.3
18.1
FreeSOLO (Wang al., 2022b)
11.6
5.9
7.3
8.0
2.2
13.8
2.4
Self-
CuLLER (Wang et al., 2023)
28.1
20.2
26.3
23.1
17.0
28.7
8.9
supervised
HASSOD (Cao al., 2023)
28.3
22.5
27.8
26.2
26.0
32.6
11.4
SOHES (Ours)
30.5
29.1
31.1
33.5
33.3
36.0
17.1
Improvement over HASSOD
+2.2
+6.6
+3.3
+7.3
+7.3
+3.4
+5.7
Reduced Gap vs. SAM
-10%
-28%
-18%
-37%
-21%
*
-85%
* SOHES outperforms SAM on PartImagenet.
EXPERIMENTS
In this section, thoroughly evaluate SOHES on various datasets and examine the ViT-based backbone improvement for downstream tasks. We perform series of ablation study experiments to demonstrate the efficacy of modules and steps SOHES. We also discuss limitations of SOHES in Appendix E. Additional qualitative results are shown in Appendix
4.1 TRAINING AND EVALUATION DATA
We train our SOHES model on the SA-IB (Kirillov et al., 2023) dataset. In SA-IB, there are 11 million images equally split into 1,000 packs. Unless otherwise specified, we use 20 packs of raw images (2%) for training, and 1 different pack (0.1%) for evaluation.
For evaluation purposes, we test SOHES on various image datasets with segmentation mask an- notations in zero-shot manner (i.e., no further training on evaluation datasets). The diversity in the evaluation datasets can simulate the challenge unseen entity classes and image domains in an open-world setting. Since the annotations in each dataset may only cover entities from pre-defined list classes, the commonly used MS-COCO style average precision (AP) metric for closed- world detection/segmeetatioo would incorrectly penalize open-world predictions that cannot be matched with ground truths in known classes. More details of the AP metric are discussed in Appendix B. Following prior work (Kim et al., 2022; Wang et al., 2022a; Liu et al., 2022; Cao et al., 2023), mainly consider the average recall (AR) metric for up to 1,000 predictions per image when compar- ing different methods. Other implementation details are in Appendix A.
4.2 OPEN-WORLD ENTITY SEGMENTATION
We evaluate SOHES variety of datasets, including MS-COCO (Lin et al., 2014), LVIS (Gupta et al., 2019), ADE20K (Zhou et al., 2017), EntitySeg (Qi et al., 2023), and SA-1B (Kirillov et al., 2023). These datasets include natural images of complex scenes, in which multiple visual entities of diverse classes present and are labeled with segmentation masks. Thus, the collection of such evaluation datasets can faithfully reflect the performance of an open- open-world segmentation model. We compare with recent self- self-supervised methods FreeSOLO (Wang et al., 2022b), CutL.RR (Wang et al., 2023), and HASSOD (Cao al., 2023). We aim to close the gap between self-supervised methods and the supervised state-of-the-art model SAM (Kirillov et al., 2023). Notably, we use only 2% images as SAM for training SOHES, and we do not require any human annotations on these images.
As summarized in Table and Table 4, SOHES consistently outperforms the prior state-of-the- art HASSOD by large margins (e.g.. +7.3 AR on SA-1B and EntitySeg). Mcanwhile, SOHES significantly closes the gap between self-supervised methods and supervised methods. For instance, using only 2% unlabeled data in SA-1B, SOHES already achieves over half AR of SAM. SOHES also reduces the gap between self-supervised methods and SAM on SA-1B relatively by 37%.
4.3 PART SEGMENTATION
In additional to whole entities, SOHES also learns to segment their constituent parts and subparts. To evaluate our hierarchical segmentation results, compare them with the ground-truth mask

---

Published as a conference paper at ICLR 2024
Figure 6: Downstream performance of ViT- based backhones. We freeze the ViT and fine- tune a ViT-Adapter and lightweight segmen- tation/detection head on ADE20K/MS-COCO. The backbone further fine-tuned in SOHES is more adapted to dense-prediction tasks.
Mask AR
Resolution 224
Resolution 448
16 Patch size
Figure 7: Mask quality of the initial pseudo- labels produced by DINO backbones with different pre-training configurations. A small pre-training patch size leads to better fine- grained features and high-quality pseudo-labels.
Backbone
ADE20K mloU
MS-COCO AP
Pre-trained by DINO
35.2
22.7
Fine-tuned by SOHES
39.6
24.4
annotations of object parts (e.g., heads and tails of animals) in two datasets, PartImageNet (He et al., 2022a) and PACO-LVIS (Ramanathan et al., 2023), and summarize the results in Table and Table 5. Compared with prior self-supervise baselines, SOHES more accurately localizes meaningful parts of entities, and almost doubles CutLER's 's performance on PACO (8.9 AR - 17.1 AR). Impressively, SOHES outperforms SAM on PartImageNet and is par with SAM on PACO. The reason is that SOHES can predict more parts and subparts than SAM (see Figure 14), which are the focus of the two datasets' annotations.
4.4 IMPROVED BACKBONE FEATURES
Through our self-instruction and correction phases, we adapt self-supervised representation DINO to open-world segmentation model. Consequently, our fine-tuned visual backbone can better fit into other dense prediction tasks. To test such abilities, we compare a) the ViT-B/8 backbone pre- trained by DINO and b) the backbone further tuned SOHES, in downstream tasks of semantic segmentation on ADE20K and object detection on MS-COCO. The downstream fine-tuning is per- formed in a minimalistic style, mimicking the linear probing (Chen et al., 2020; He al., 2020) in self-supprvised representation learning. For semantic segmentation, we directly attach a linear classifier on the feature maps from ViT-Adapter; for object detection, we attach the simplest Reti- naNet (Lin et al., 2017) detection head on the ViT-Adapter. We keep the ViT parameters frozen during the supervised fine-tuning. Table 6 summarizes the results, demonstrating that SOHES can adapt the ViT-based backbone to generate better features for dense prediction downstream tasks.
4.5 ABLATION STUDY
In this subsection, we ablate the design choices in SOHES on SA-1B, and provide our insights for future research in self-super open-world segmentation. Further details about the choices of hyper-parameters and model architectures are discussed in Appendix D.
DINO backbone. In recent self-supervised object localization/discoverr work (Siméoni et al., 2023; Wang et al., 2023), researchers prefer the ViT backbone pre-trained by DINO, in particular ViT-B with patch size& 8. We have also observed that a ViT backbone with patch size 8 leads to better mask quality in SOHES. To investigate this, we use DINO to pre-train ViT-B backbones with varying patch size and input resolution configurations, with shorter 100-epoch training schedule (DINO originally pre-trains on ImageNet (Deng et al., 2009) for 300 epochs). Then, we repeat our self- exploration phase with these backbones, and the resulting mask quality comparison summarized in Figure 7 and Table 7. From this comparison, we can observe that the small patch size is positively correlated with the mask quality. When the patch size decreases from 32 to 8, the AR significantly improves. Meanwhile, the input resolution does not influence the mask quality as much. The small patch size may better support the ViT capture pixel-aligned details for localizing entites, and thus is more suited in our self-supervised segmentation task. It is worth noting that we cannot further reduce the patch size due to computational constraints. Whenever the patch size halved, ViT needs to process 4x patches, and perform 16x computation in self-attention Therefore, the off-the-shelf DINO ViT-B/8 is the best choice in our task.
Steps in self- eeff-xxploration. In our self-exploration phase, we have delicately designed series of steps to generate, select, and refine the labudo-labels. We summarize the impact of the design choices in Table 2. In the first global clustering step, if we one fixed merging threshold 6 m larger m leads to more masks per image and better coverage of entities (increasing AR), and also introduces noises (oscillating AP). We choose mix the results with different thresholds together
8

---

Published as a conference paper at ICLR 2024
Table 2: Impact of each step in self-exploration. We mix the global clustering results from mul- tiple merging thresholds, adopt the local re-clustering, and use the off-the-shelf CascadePSP mask refinement module to obtain the best initial pseudo-labels.
Time/Img (scc)
Mask Quality
Step
Choice
Masks/Img
AP
ARg
ARar
AR2
AR
Onerge 0.1
4.9
0.2
0.0
0.0
0.3
0.1
meerge =
3
4.9
0.8
0.1
0.1
0.6
0.3
Omerge 0.3
9
4.5
0.9
0.4
0.6
2.0
1.0
merrge =
23
3.8
0.7
1.5
5.4
meerge
58
2.7
1.2
3.4
11.1
5.4
Omerge 0.6
131
2.2
0.6
1.5
6.0
15.3
8.1
Mix w/ NMS
148
5.3
1.9
6.5
17.2
w/ local re- re-clustering
115
8.4
2.0
5.1
10.1
17.5
11.6
DenseCRF (Krähenbühl & Koltun, 2011)
61
18.2
4.7
3.5
9.5
20.7
12.0
CRM (Shen al., 2022)
71
18.7
2.7
4.7
13.5
20.2
14.1
CascadePSP (Cheng al., 2020)
101
15.2
4.7
6.0
15.8
22.6
16.4
Table 3: Impact of the dynamic threshold in self-correction. If the vanilla teacher-student learning employed in the self-correction phase (row 2), the imbalance between small and large entity segmentation intensified which leads to worse overall performance. Our dynamic threshold for filtering the teacher's s pscudo-labels (row 3) can encourage the student's predictions for small and medium entities and improve the overall AR.
Mask Quality
Training
AP
ARs
ARM
ARt
AR
Phase
12.8
8.0
33.7
43.0
32.6
Phase + w/o dynamic threshold
10.9
6.3
32.5
45.3
32.5
Phase + w/ dynamic threshold
12.9
8.6
35.2
42.0
33.3
and remove duplicates, which provides the best AR and only slightly increases the number of masks compared with the largest mmerge In the second local re-clustering step, we significantly improve the recall for small entities, relatively by 168%. This step ensures that our model receives adequate supervision from small entities. We also improve the overall recall by 2.5 AR. Finally, in the third refinement step, we adopt CascadePSP (Cheng al., 2020) because can best boost the overall mask quality. The other two options, DenseCRF (Krähenbühl & Koltun, 2011) and CRM (Shen et al., 2022), are also viable, but they change the pseudo-labels more aggressively, leading to removal of many potential entities. Overall, each step in our self-exploration phase contributes to the high-quality initial pseudo-labels for SOHES. Notably, we can parallelize the processing for each image and accelerate self- ceff-exploration with more compute nodes.
Self-correction. In our self-correction phase, we adopt a teacher-student mutual-learning frame- work from semi-supervised learning, continuously improve the segmentation model by itself. However, as shown Table 3, the initial attempt the mutual-learning with a fixed confidence threshold leads to procedure, as indicated by the decreased AR for small and medium entities and increased AR for larger entities. Therefore, we need a dynamic threshold that allows more small and medium pseudo- labels from the teacher model, and balances the student's 's prediction for entities of different scales. With the dynamic threshold, we can improve ARş and ARM relatively by 7.5% and 4.5%, with an acceptable cost of 2.3% AR Consequently, the overall AR is improved by 0.7.
performance. In fact, the imbalanced distribution is reinforced during this
CONCLUSION
We present SOHES, self-supervised approach towards open-world entity segmentation with hi- erarchical structures. Through three phases of self-evolution, a self-supervised learner is adapted to open-world segmentation model. By recognizing and localizing entities and their constituent parts in an open world with superior mask quality, SOHES substantially closes the gap between self-supervised and supervised methods, and sets the state of the art various datasets.

---

Published as a conference paper at ICLR 2024
Acknowledgement. This work supported in part by NSF Grant 2106825, NIFA Award 2020- 67021-32799, and the Jump ARCHES endowment through the Health Care Engineering Systems Center. This work used NVIDIA GPUs NCSA Delta through allocations CIS220014, CIS230012, and CIS230013 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Sup- port (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, 2018.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In CVPR, 2015.
Abhijit Bendale and Terrance Boult. Towards open set deep networks. In CVPR, 2016.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In CVPR, 2018.
Shengcao Cao, Dhiraj Joshi, Liang-Yo Yan Gui, and Yu-Xiong Wang. HASSOD: Hierarchical adaptive self-supervised object detection. In NeurIPS, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Liang-Chies Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, (00(4):844-84 2017.
2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2022.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. CascadePSP: Toward class- agnostic and very high-resolution segmentation via global and local refinement. In CVPR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. In WACV, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: dataset for large vocabulary instance seg- mentation. In CVPR, 2019.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome Friedman. The elements of statistical learning: Data mining, inference, and prediction. Springer, 2009.
10

---

Published as a conference paper at ICLR 2024
Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. PartlmagNNet: A large, high-quality dataset of parts. In ECCV, 2022a.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022b.
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning segment every thing. In CVPR, 2018.
Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Premkumar Natarajan. Class-agnostic object de- tection. In WACV, 2021.
K Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In CVPR, 2021.
Tarun Kalluri, Weiyao Wang, Heng Wang, Manmohan Chandraker, Lorenzo Torresani, and Du Tran. Open-world instance segmentation: Top-down learning with bottom-up supervision. arXiv preprint arXivv2303.05500 2023.
Dahun Kim, Tsung-Yi Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open- world object proposals without learning to classify. IEEE Robotics and Automation Letters, 7(2): 5453-5460, 2022.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. .ngeting... In ICCV, 2023.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected CRFs with Gaussian edge potentials. In NeurlPS, 2011.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Yang Liu, Idil Esen Zulfikar, Jonathon Luiten, Achal Dave. Deva Ramanan, Bastian Leibe, Aljoša Ošep, and Laura
Opening up open world tracking. In CVPR, 2022.
Yen-Cheng Liu, Chih- Yao Ma, Zijian He, Chia-Wer Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In ICLR, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao Philip Torr, Zhe Lin, and Jiaya Jia. Open-world entity segmentation. TPAMI, 45(7):8743-8756, 2022.
Lu Qi, Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Weidong Guo, Jiaya Jia, Zhe Lin, and Ming- Hsuan Yang. High-quality entity segmentation. In ICCV, 2023.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi. Yiwen Song, Abhi- manyu Dubey, and Dhruv Mahajan. PACO: Parts and attributes of common objects. In CVPR, 2023.
11

---

Published as a conference paper at ICLR 2024
Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance Boult. Toward open set recognition. TPAMI, 35(7)11757-177,, 2012.
Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, 2022.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. TPAMI, 22(8):888 905, 2000.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC, 2021.
Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobecký, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. In CVPR, 2023.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
consis-
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 104:154-171, 2013.
Huy V Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce. Unsu- pervised image matching and object discovery as optimization. In CVPR, 2019.
Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised, multi-object discovery in large-
scale image collections. In ECCV, 2020.
Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsuper- vised object discovery. In NeurlPS, 2021.
Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In ICCV, 2021.
Weiyao Wang, Matt Feiszli, Heng Wang, Jitendra Malik, and Du Tran. Open-world instance seg- mentation: Exploiting pseudo ground truth from learned pairwise affinity. In CVPR, 2022a.
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. FreeSOLO: Learning to segment objects without annotations. In CVPR, 2022b.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR, 2023.
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. TokenCut: Segmenting objects in images and videos with supe-uupere transformer and normalized cut. In CVPR, 2022c.
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208 06677, 2022.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. GiTupViT: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
Self-exploration. We use DINO (Caron et al., 2021) pre-trained ViT-B/8 the feature extractor to generate patch-level visual features. During the global clustering step, we first resize unlabeled images to resolution S x 5 = 1, 024 x 1, 024, and cluster x patches with merging thresholds {0.6, 0.5, 0.4, 0.3, 0.2, 0.1), which are decided based the number of pscudo- labels (see Figure 8). Then in the local re-clustering step, we further investigate regions smaller than Osmull % = 1 /1 024 of the total image area, and crop local images around them. The local image is resized to 5" x S' = 256 x 256 and its subregions are clustered. In the next step, we use the off- the-shelf CascadePSP (Cheng et al., 2020) model refine the masks, In the final step of hierarchy analysis, the coverage threshold is set to Oowerr % 90%.
(Omorge. I Jt=]
Self-instruction. We learn segmentation model composed of DINO (Caron et al., 2021) pre- trained ViT-B/8, ViT-Adapter (Chen et al., 2022), Mask2Former (Cheng et al., 2022), and our an- cestor prediction head. The model is trained on 8 compute nodes, each equipped with 8 NVIDIA A100 GPUs. The total batch size is 128, and the number of training steps is 40,000. We optimize the model with the Adan optimizer (Xie et al., 2022) and base learning rate of 0.0008. The total training time is about 3 days.
Self-correction.
The teacher-student mutual learning starts after the ceff-instruction phase. trains the model for additional 5,000 iterations. The teacher is updated as the exponential moving average of the student, with momentum m = 0.9995. The loss terms from the initial pseudo-labels and the teacher's 's pscudo-labels are weighted equally. In the dynamic threshold, set Sscore. large 0.7, Score, small = 0.3, y = 200.
B DEFICIENCY OF AP METRIC IN OPEN-WORLD SEGMENTATION
The MS-COCO (Lin et al., 2014) style average precision (AP) is a prevalent metric for evaluating object detection and instance segmentation models in the traditional closed-world setting with pre-defined scope of categories. However, for the open-world segmentation task, the AP metric becomes misleading and cannot accurately reflect the open-world model's true performance: When evaluating apen-world segmentation models (which try to segment "everything") on datasets with plosedworld annotations (such MS-COCO/LVIS which only include pre-defined, limited set of entity classes), AP would penalize model predictions that are actually valid entities, but just not annotated by the dataset.
Consequently, when the dataset annotations cannot cover all entities, AP becomes misleading for judging the performance of open-world models. As example, the AP of SAM (Kirillov al., 2023) is lower than CutLER (Wang et al., 2023) on MS-COCO (6.1 vs. 9.8, Table 4) and PartIm- ageNet (3.4 vs. 5.3. Table 5), which definitely cannot imply SAM is a model inferior CutLER. The lower AP of SAM is due to insufficient annotations in these two datasets, not the capability of SAM. Meanwhile, on datasets which explicitly try to mitigate these annotation limitations like SA-1B (Kirillov et al., 2023), AP can be more reliable than on traditional closed-wond datasets like MS-COCO (Lin et al., 2014). For instance, AP on SA- IB (the most densely annotated dataset evaluated in our work, see Table 4) matches qualitative comparison and follows the same trend as AR, and our AP (12.9) significantly surpasses CutLER's 's AP (7.8). Prior work (Cao et al., 2023) has made similar observation about the deficiency of MS-COCO AP evaluation in the context of self-supervised object detection.
In general, choose the average recall (AR) as the main metric instead of AP, because AR does not penalize valid open-world predictions. Note that this choice of AR over AP also commonly adopted in the open-world literature. Examples include (but are not limited to) detection (Kim al., 2022), segmentation (Wang et al., 2022a), and tracking (Liu et al., 2022).
C ADDITIONAL EVALUATION RESULTS
We provide additional evaluation results with more metrics in Table and Table 5, as a supplement to Table in the main paper.
13

---

Published as a conference paper at ICLR 2024
Table 4: Detailed zero-shot evaluation results on image datasets with annotations of whole entities.
Mask Quality
Method
AP
ARş
ARM
AR,
AR
MS-COCO (Lin et al., 2014)
SAM (Kirillov et al., 2023)
33.4
59.6
64.1
49.6
FreeSOLO (Wang et al., 2022b)
4.3
0.5
11.5
31.2
11.6
CutLER (Wang et al., 2023)
9.8
13.1
31.6
49.3
28.1
HASSOD (Cao et al., 2023)
6.0
14.0
34.1
45.2
28.3
SOHES (Ours)
2.1
19.8
31.0
48.8
30.5
LVIS (Gupta et al., 2019)
SAM (Kirillov et al., 2023)
6.7
31.1
71.3
74.6
46.1
FreeSOLO (Wang et al., 2022b)
1.9
0.2
9.2
31.7
5.9
CutLER (Wang et al., 2023)
3.6
11.3
31.1
46.2
20.2
HASSOD (Cao et al., 2023)
4.2
12.7
36.1
47.8
22.5
SOHES (Ours)
1.9
19.8
39.4
59.2
29.1
ADE20K (Zhou et al., 2017)
SAM (Kirillov et al., 2023)
7.8
31.6
59.2
62.5
45.8
FreeSOLO (Wang et al., 2022b)
2.3
0.5
9.3
27.1
7.3
CutLER (Wang et al., 2023)
5.2
15.3
34.7
44.5
26.3
HASSOD (Cao et al., 2023)
7.0
16.2
36.7
46.7
27.8
SOHES (Ours)
2.6
21.8
37.2
49.0
31.1
EntitySeg
(Qi et al.,
2023)
SAM (Kirillov et al., 2023)
14.8
11.0
25.0
55.6
45.9
FreeSOLO (Wang al., 2022b)
3.0
0.1
1.2
10.7
8.0
CutLER (Wang et al., 2023)
7.7
6.1
15.3
27.2
23.1
HASSOD (Cao et al., 2023)
6.1
7.6
20.1
30.2
26.2
SOHES (Ours)
5.0
9.1
21.6
39.6
33.5
SA-IB (Kirillov al., 2023)
SAM (Kirillov et al., 2023)
38.9
20.0
59.9
82.2
60.8
FreeSOLO (Wang al., 2022b)
1.5
0.0
0.2
6.9
2.2
CutLER (Wang et al., 2023)
7.8
4.9
13.9
28.5
17.0
HASSOD (Cao et al., 2023)
13.8
12.9
22.8
38.3
26.0
SOHES (Ours)
12.9
8.6
35.2
42.0
33.3
Table 5: Detailed zero-shot evaluation results on image datasets with annotations of object parts.
Mask Quality
Method
AP
ARs
ARM
ARt
AR
PartlmageNer (He et al., 2022a)
SAM (Kirillov et al., 2023)
3.4
25.4
29.3
28.5
28.3
FreeSOLO (Wang et al., 2022b)
3.3
0.6
7.7
26.4
13.8
CutLER (Wang et al., 2023)
5.3
13.2
26.5
38.1
28.7
HASSOD (Cao et al., 2023)
4.5
19.0
32,9
38.7
32.6
SOHES (Ours)
1.2
30.3
32.4
42.3
36.0
PACO-LVIS (Ramanathan et al., 2023)
SAM (Kirillov et al., 2023)
1.0
11.9
34.6
41.1
18.1
FreeSOLO (Wang et al., 2022b)
0.2
0.1
5.8
21.3
2.4
CutLER (Wang et al., 2023)
0.2
5.0
18.7
25.1
8.9
HASSOD (Cao et al., 2023)
0.4
6.4
24.2
31.4
11.4
SOHES (Ours)
0.4
12.0
29.0
41.9
17.1
14

---

Published as a conference paper at ICLR 2024
D ADDITIONAL ABLATION STUDY RESULTS
Since SOHES is self-supervised approach, we do not base our selection of hyper-paramettrs posteriori model performance. Instead, we choose hyper-paramettrs by considering simple criteria such computation constraints small-scale experiments. In this section, detail the design choices in SOHES.
Merging thresholds. In the first step of self-exploration phase, cluster patches into coherent regions and stop at pre-set merging thresholds meerge.1 > ... > mmerge, We choose these thresholds based a practical computation constraint: pseudo-labels per image. When there are too many pseudo-labels, data loading, augmentation, and pre-processing would become bottleneck in model training. Therefore, we control the number of pseudo-labels per image be under 200. As shown in Figure 8, the quantity of pseudo-labels grows significantly when the merging threshold is larger. In order not to exceed 200 masks per image, we set the merging thresholds 0.6, 0.5, 0.4, 0.3, 0.2, 0.1.
300
Pseudo-labels per image
250
Figure 8: Relation between the number of pseudo-labels and the merging thresholds. As the merging threshold increases, the number of pseudo-labels rapidly grows. To control the number of pseudo-labels per image, we choose merging thresholds no larger than 0.6.
200
150
100
50
Merging threshold
Local re-clustering threshold. the second step of self-exploration, we use threshold 0sman % to select small regions that require local re-examination. Pseudo-labels with areas smaller than 8maal % of the whole image are processed via local re-clustering, and this step improves the coverage of small entities. We choose small % 1/ 11, 024 because 1) the commonly adopted MS-COCO style evaluation (Lin et al., 2014) defines small objects as objects whose area is smaller than 32 322 pixels; 2) images resized 1, 024 1, 024 in the self-exploration phase; 3) (32 32)/(1, 024 x 1, 024) = 1/1, 024. In fact, if the threshold 0emall% % is larger, the coverage of small entities could not be significantly improved further, and the processing time would be longer, as shown in Table 6a.
Table 6: Impact of hyper-parameters 0mall % and O cover %.
(a) Choice of the local re-clustering threshold Osmall We set Osmall 1/1, 024 by considering the relative areas f small entities. Using larger 6 % does not further improve the coverage of small entities, but introduces additional com- putation overheads.
(b) Choice of the coverage threshold Ócover %. set 9eoer = 0.9 for robust hicrarchi- cal relations between our pscudo-labels. When Oewer % varies within [0.6, 0.95), the hicrarchi- cal relations stable.
i 8amall %
ARş
Time/Img (sec)
Relative Change
Ocover %
W.r.t. Ozover 0.9
0 (No local re-clustering) 1/2, 048
1.9
0.0
4.3
6.0
0.6
9%
1/1, 0)24
5.1
8.4
0.7
7%
1/512
5.1
10.0
0.8
4%
0.85
2%
0.9
0%
0.95
4%
1.0
36%
Coverage threshold. In the final step of self-exploration, we analyze the hierarchical relations be- tween pseudo-labels. The pairwise test involves a coverage threshold Ocver %. For robustness,
15

---

Published as a conference paper at ICLR 2024
choose Beover% % 0.9 to allow ancestor pseudo-label to not necessarily cover all of its descen- dants' pixels. Indeed, when Beoerr E [0.6, 0.95], the induced hierarchical relations are relatively stable, as shown in Table 6b.
DINO backbone. As a supplement to Figure in the main paper. Table summarizes the detailed statistics of the mask quality of initial pseudo-labels produced by DINO backbones pre-trainn me-trained with different configurations. also includes the mask quality after each self-exploration step.
Table 7: Comparison of DINO backbones pre-trained with different configurations. A small pre-training patch size is critical for producing fine iieeggained visual features and high-quality pseudo- label masks. The steps 1, 2, and 3 refer to global clustering, local re-clustering, and mask refinement, respectively.
Pre-training
Mask Quality
Step
Masks/Image
Resolution
Patch Size
AP
ARg
ARM
AR,
AR
1
144
0.8
1.6
5.3
14.7
7.6
224
8
2
124
1.3
4.1
7.8
15.0
9.4
3
105
3.2
5.0
12.6
20.6
13.8
108
0.8
1.3
4.5
13.0
6.6
224
16
2
75
1.7
3.5
6.9
13.3
8.3
3
65
4.0
4.2
11.3
17.0
11.9
1
43
0.6
0.7
2.6
5.8
3.3
224
32
2
17
0.9
0.3
2.2
5.9
3.0
3
14
1.8
0.4
3.4
6.6
3.9
1
114
1.0
1.2
4.2
14.6
6.9
448
8
2
103
1.6
3.1
6.4
14.8
8.5
3
82
3.4
3.9
10.8
19.1
12.2
1
102
0.8
1.3
3.9
13.1
6.3
448
16
2
84
1.5
3.8
6.7
13.4
8.3
3
3.8
4.6
10.8
17.1
11.8
0.7
2.2
7.1
3.5
448
32
2
25
1.0
1.1
2.5
7.2
3.7
3
20
2.4
1.4
4.1
8.3
4.9
Segmentation head. SOHES, we choose Mask2Former (Cheng et al., 2022) as our segmenta- tion head mainly for making hierarchical predictions. In Cascade Mask R-CNN (Cai & Vasconcelos, 2018) used by prior work (Wang et al., 2023; Cao et al., 2023), each proposal is predicted indepen- dently, and therefore, analyzing hierarchical relations between entities would be challenging. In contrast, the attention modules in Mask2Former allow information exchange among queries, so we can build our ancestor prediction head Mask2Former (see Figure 4). For a more comprehensive comparison with prior work, we train segmentation model based on Cascade Mask R-CNN using SOHES without the entity hierarchy. As shown Table this model still significantly outperforms CutLER and HASSOD with the same segmentation head.
When using the Mask2Former segmentation head, we can extend it with our proposed ancestor pre- diction head to perform the additional task of hierarchical relation predictions. This module can be considered as an add-on to the original segmentation head with minimal influence on the mask quality, since this ancestor prediction head operates in parallel to the mask prediction head. In fact, the segmentation performance of a SOHES model trained without the ancestor prediction head close to that of the standard SOHES model (e.g., on SA-1B 33.0 Mask AR without ancestor predic- tion vs. 33.3 Mask AR with ancestor prediction), but the former model cannot predict hierarchical relations.
16

---

Published as a conference paper at ICLR 2024
Table Comparison of models trained with the Cascade Mask R-CNN segmentation head. In the main paper, we choose the Mask2Former segmentation head to model hierarchical relations be- tween predictions, while prior methods usually use Cascade Mask R-CNN. With the same Cascade Mask R-CNN segmentation head, SOHES still outperforms prior work.
Mask Quality (AR)
Method
Segmentation Head
LVIS
EntitySeg
SA-IB
CutLER (Wang et al., 2023)
20.2
23.1
17.0
Cascade Mask R-CNN
HASSOD (Cao et al., 2023)
22.5
26.2
26.0
(Cai Vasconcelos, 2018)
SOHES (Ours)
27.0
29.2
31.9
SOHES (Ours)
Mask2Former (Cheng et al., 2022)
29.1
33.5
33.3
E LIMITATIONS
In Figure 9, visualize failure cases of SOHES: 1) When there discontinuous entities or occlusion (e.g., sky separated by foreground objects), SOHES may not correctly associate the disconnected segments of the same entity. The reason is that in the self-exploration phase, we only merge adjacent regions. The model rarely observes one entity separated in multiple disconnected re- gions. We observe that the copy-paste data augmentation (Ghiasi et al., 2021) can simulate occlusion and partially mitigate this issue, so we have adopted such data augmentation in SOHES training. 2) SOHES often produces imprecise segmentation masks for letters and characters. The boundary is not perfectly aligned with strokes and the mask is often larger than the letter. 3) When there are blurry backgrounds, SOHES tends not to predict a mask for the background. Similar failure cases can be observed when applying prior methods (Wang et al., 2023; Cao et al., 2023; Kirillov et al.,
2023) to images affected by occlusion, text overlays, or blurring. We aim to resolve these issues with improved eseudo-labeling strategies in future work.
AYLT
MMIGRAZ NE NON ON RED BASTA MORTI IN QUESTUR Lial 'ees
NON
BASTA
Figure 9: Limitations of SOHES. Our method sometimes fail accurately segment discontinuous or occluded entities (left), letters or characters (middle), and blurred background (right).
F QUALITATIVE RESULTS
In Figures 10, 11, 12, 13, and 14, we visualize the segmentation results of SOHES, and qualitatively compare SOHES with the supervised model SAM on the evaluation datasets we have used in the main paper.
17

---

Published as a conference paper at ICLR 2024
STOP
STOP
STOP
Figure 10: Qualitative results on MS-COCO images. In each group of images, from top to bottom show the original input image, segmentation SAM (Kirillov al., 2023), and segmentation by SOHES. As a self-supervised method, SOHES can achieve results that are comparable to those produced by the supervised model SAM.
18

---

Published as a conference paper at ICLR 2024
Figure 11: Qualitative results on ADE20K images. In each group of images, from top to bottom show the original input image, segmentation by SAM (Kirillov al., 2023), and segmentation by SOHES. As a self-supervised method, SOHES can achieve results that are comparable to those produced by the supervised model SAM.
19

---

Published as a conference paper at ICLR 2024
Figure 12: Qualitative results on EntitySeg images. In each group of images, from top to bottom show the original input image, segmentation by SAM (Kirillov al., 2023), and segmentation by SOHES. As a self-supervised method, SOHES can achieve results that are comparable to those produced by the supervised model SAM.
20

---

Published as a conference paper at ICLR 2024
Metro
Plaza de ESPDAR
Eapada
Qualitative results on SA-1B images. In each group of images, from top to bottom show the original input image, segmentation by SAM (Kirillov al., 2023), and segmentation by SOHES. As a self-supervised method, SOHES can achieve results that are comparable to those produced by the supervised model SAM.
Figure 13:
21

---

Published as a conference paper at ICLR 2024
Figure 14: Qualitative results on PartImageNet images. each group of images, from top to bottom we show the original input image, segmentation by SAM (Kirillov et al., 2023), and segmen- tation by SOHES. As a self-supervised method, SOHES can achieve results that are comparable to those produced by the supervised model SAM.
22


Reviews:
[{"review_id": "p86IwNLQeH", "reviewer": "Reviewer_PuvF", "summary": "The authors introduced the Self-supervised Open-world Hierarchical Entity Segmentation (SOHES) method, a three-phase approach for entity segmentation. The first phase, Self-exploration, uses a pre-trained DINO model to produce initial pseudo-labels. By clustering visual features, it identifies regions representing meaningful entities. In the Self-instruction phase, a Mask2Former segmentation model refines the segmentation by training on these initial labels. Even with some label noise, the model effectively averages out inconsistencies, resulting in better mask predictions. The final Self-correction phase uses a teacher-student mutual-learning framework to further refine the model's predictions and adapt to open-world segmentation. This approach only uses raw images without human annotations. A standout feature of SOHES is its ability to segment not just whole entities but also their parts and sub-parts.", "questions": "My main questions are listed in the weakness section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "[Task] Unsupervised image segmentation holds significant importance, and this study successfully performs segmentation without human supervision, offering segmentation masks at multiple levels of granularity.\n\n[The generation of hierarchical masks] The approach to generate unsupervised hierarchical masks is pretty interesting. And surprisingly, this method surpassed SAM in recall on some evaluation benchmarks. \n\n[Paper writing] The paper is well-articulated, effectively communicating the central ideas.", "weaknesses": "[Technical Contributions] The three phases proposed in this work are very similar to the Cut-and-Learn pipeline proposed by CutLER [1]. Self-exploration is pretty similar to the MaskCut stage in CutLER, which also leverages DINO feature for pseudo-label generation. Self-instruction is the same as the LEARN process of CutLER, which trains a model on pseudo-labels. And, the Self-correction stage can be viewed as a variant of CutLER's multi-round self-training, but with a teacher-student framework. I agree that while there are some implementation differences between the stages in SOHES and CutLER, their core concepts are largely analogous.\n\n[Model performance] SOHES performs much worse than CutLER (9.8 vs. 2.1 on COCO and 3.6 vs. 1.9 on LVIS) in terms of the mask AP. This works show stronger performance than the previous SOTA CutLER and SAM on some benchmarks, however, the main results are AR (averaged recall). Recall is important, however, for many downstream tasks, the AP is still the most valuable evaluation metric. \n\n[Unfair comparison] The main baseline CutLER used Cascade Mask RCNN as the segmentation model, while this work used Mask2Former, a stronger segmentation model, as the base model. This makes the performance comparison unfair. \n\n[1] Wang, Xudong, Rohit Girdhar, Stella X. Yu, and Ishan Misra. \"Cut and learn for unsupervised object detection and instance segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3124-3134. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kw8jtsEZCj", "reviewer": "Reviewer_vWUT", "summary": "The paper introduces the Self-supervised Open-world Hierarchical Entity Segmentation (SOHES) approach for computer vision, which can segment entities in images beyond pre-defined classes without human annotations. SOHES uses three stages: self-exploration, self-instruction, and self-correction, generating high-quality pseudo-labels from visual feature clustering and refining them through mutual-learning. This method achieves a new standard in self-supervised open-world segmentation, eliminating the need for human-annotated masks.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe task SOHES is seldom investigated.\n2.\tThis paper proposed a new method to generate hierarchical proposals.\n3.\tThis paper proposed an ancestor prediction head, which is novel.\n4.\tThe proposed method significantly outperformed the previous methods.", "weaknesses": "1.\tAlthough this paper divide the stages into self-exploration, self-instruction, and self-correction. But it looks like previous papers[1] that generate pseudo-labels, then training from pseudo-labels, and apply self-training to improve the model. The framework is actually quite common. So what is the core difference from the previous works in the framework?\n2.     The authors claimed that “Existing segmentation models cannot predict the hierarchical relations among masks. ” However, methods like Groupvit[2] already can predict the hierarchical relations among masks.\n3.\tThe proposed method introduced too many hyperparameters such as Theta_{merge}, Theta_{small}, Theta_{cover}. There is not explanation for the selection of some hyperparameters, such as Theta_{cover}. \n4.\tAlthough the authors proposed the ancestor prediction head for the hierarchical segmentation. However, the mask prediction of masks at each hierarchy is independent.  I am curious if the relation modeling of hierarchy can contribute to better mask predictions. I am also curious about the effect of just predicting the hierarchical relationships. It seems that there is no ablation to verify if the ancestor prediction head can bring the performance improvements.\n\n[1] Wang, Xinlong, et al. \"Freesolo: Learning to segment objects without annotations.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Xu, Jiarui, et al. \"Groupvit: Semantic segmentation emerges from text supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyB0vjpqB0", "reviewer": "Reviewer_Rono", "summary": "The paper introduces SOHES, a method designed to eliminate the need for manual annotation and facilitate the learning of hierarchical associations for open-world entity segmentation. Specifically, the authors have devised a novel approach for generating initial pseudo-labels and subsequently enhancing the segmentation model through their utilization. When compared to previous research on open-world entity segmentation, SOHES demonstrates a significant reduction in the gap between self-supervised methods and the supervised SAM.", "questions": "1. Regarding the generation of pseudo-labels, I am inquisitive about why the visual feature of merged patches is computed as the sum of original features rather than an average or any other operation?\n2. Concerning the second local re-clustering step, I'm curious about whether it has an impact on the results for large entities and how the threshold for small regions affects the final results. \n4. In the ablation study, there is no information regarding how the ancestor prediction head contributes to the final results. Further analysis of the hierarchical architecture is needed for clarification.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is highly motivated, and the results of SOHES demonstrate tremendous potential in open-world entity segmentation, even surpassing SAM's performance on certain datasets. \n2. The paper is excellently structured and provides a clear and easy-to-follow presentation.", "weaknesses": "1. The analysis of the hierarchical architecture is inadequate.\n2. Certain details in the method lack clarity, and there is a noticeable absence of some ablation experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)