Hierarchical Open-vocabulary Universal Image Segmentation
Authors: Xudong Wang
Shufan Li
Konstantinos Kallidromitis
Yusuke Kato
Kazuki Kozuka
Trevor Darrell
Venue: NeurIPS 2023 poster (2023)
Keywords: Universal Image Segmentation
Hierarchical
Open-vocabulary

Abstract: Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions. However, complex visual scenes can be naturally decomposed into simpler parts and abstracted at multiple lev4 els of granularity, introducing inherent segmentation ambiguity. Unlike existing methods that typically sidestep this ambiguity and treat it as an external factor, our approach actively incorporates a hierarchical representation encompassing different semantic-levels into the learning process. We propose a decoupled text-image fusion mechanism and representation learning modules for both “things” and “stuff”. Additionally, we systematically examine the differences that exist in the textual and visual features between these types of categories. Our resulting model, named HIPIE, tackles HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a unified framework. Benchmarked on diverse datasets, e.g., ADE20K,COCO, Pascal-VOC Part, and RefCOCO/RefCOCOg, HIPIE achieves the state-of14 the-art results at various levels of image comprehension, including semantic-level (e.g., semantic segmentation), instance-level (e.g., panoptic/referring segmentationand object detection), as well as part-level (e.g., part/subpart segmentation) tasks.

Hierarchical Open-vocabulary Universal Image
Segmentation
Xudong Wang
Shufan Li
Konstantinos Kallidromitis"
Yusuke Kato²
Kazuki Kozuka²
Trevor Darrell'
Berkeley AI Research, UC Berkeley
Paaasonicc AI Research
project page:
Abstract
Open-vocabulary image segmentation aims to partition an image into semantic regions according arbitrary text descriptions. However, complex visual scenes can be naturally decomposed into simpler parts and abstracted at multiple lev- els of granularity, introducing inherent segmentation ambiguity. Unlike existing methods that typically sidestep this ambiguity and treat it as an external factor, our approach actively incorporates a hierarchical representation encompassing different semantic-levels into the learning process. We also propose a decoupled text-image fusion mechanism and representation learning modules for both "things" and "stuff" Additionally, we systematically examine the differences that exist in the textual and visual features between these types of categories. Our resulting model, named HIPIE, tackles Hlerarchical, oPen-vocabular and unIv Ersal seg- mentation tasks within a unified framework. Benchmarked on over 40 datasets, e.g., ADE20K, COCO, Pascal-VOC Part, RefCOCC/Re/COCCC ODinW and SeginW, HIPIE achieves the state-of-the-an results at various levels of image comprehen- sion, including semantic-level (e.g., semantic segmentation), instance-level (e.g., panoptic/referring segmentation and object detection), as well as part-level (e.g.. part/subpart segmentation) tasks.
Introduction
Image segmentation is a fundamental task in computer vision, enabling a wide range of applications such as object recognition, scene understanding, and image manipulation [50, 14, 42, 7, 37]. Recent advancements large language models pave the way open-vocabulary image segmentation, where models can handle a wide variety of object classes using text prompts. However, there is no single "correct" way to segment an image. The inherent ambiguity in segmentation stems from the fact that the interpretations of boundaries and regions within image depend on the specific tasks.
Existing methods for vocabblcyyy image segmentation typically address the ambiguity image segmentation by considering it as an external factor beyond the modeling process. In contrast, we adopt different approach by embracing this ambiguity and present HIPIE, as illustrated in Fig. 1, a novel Hlerarchical, oPen-vocabulary and unIvErsal image segmentation and detection model. This includes semantic-level segmentation, which focuses on segmenting objects based on their semantic meaning, as well as instance-level segmentation, which involves segmenting individual instances of objects or groups of objects (e.g., instance and referring segmentation).
'The terms things (countable objects, typically foreground) and stuff (non-object, non non-countable, typically background) [1] are commonly used to distinguish between objects that have well-defined geometry and are countable, e.g. people, cars, and animals, and surfaces regions that lack fixed geometry and are primarily identified by their texture and/or material, e.g. the sky, road, and water body. *; equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Text Prompt
Model Inference
Panoptic Sagment Dataction:
Segmants
"person, aby. free, buikding.....
"penson head, leg, arm ..."
HIPIE
Image
Referring Segment: "beach umbrella with white pole
Subpart Segment:
pperson mouth, cat neck...
Hierarchy
Person
Cat
Head
Torso
-5y
Sar
SHr
Nock
Whole
Part
Subpart
Referring Segmentation
Raw Image
SAM
Grounded SAM
HIPIE + SAM
Figure 1: HIPIE is unified framework which, given an image and set of arbitrary text descriptions, provides hierarchical semantic, instance. part, and subpart-level image segmentations. This includes open- vyeablocab semantic g-, crowds and sky), instance/panoptic (e.g., person and cat), part (e.g., head and torso), subpart (e.g., car and nose) and referring expression (e.g., umbrella with white pole) masks. HIPIE outperforms previous methods and established new SOTAs on these tasks regardless of their granularity or task specificity. Bottom images: our method can seamlessly integrate with SAM to enable class-aware image segmentation on SA-1B.
Visual Features (MAE)
Textual Features (BERT)
Additionally, our model captures finer details by incorporating part-level segmentation, which in- volves segmenting object parts/subparts. By en- compassing different granularity. HIPIE allows for a more comprehensive and nuanced analysis of images, enabling richer understanding of their contents.
0.52
Pretrain
To design HIPIE, we begin by investigating the design choices for open- open-vocabulary image seg- mentation (OIS). Existing methods OIS typi- cally adopt a text-image fusion mechanism, and employ shared representation learning module for both stuff and thing classes [4, 62, 58, 10, 56]. Fig. 2 shows the similarity matrics of vi- sual and textual features between stuff and thing classes. On this basis, we can derive several conclusions:
thing
Figure 2: Noticeable discrepancies exist the between- class similarities visual and textual features between stuff and thing classes. We propose a decoupled repre- sentation learning approach that effectively generates more discriminative visual and textual features. We ex- tract similarity matrices for the visual features, obtained through a pretrained MAE [17] or our fine-tuned one, and for the text features, produced using a pretrained BERT [6] or fine-tuned one. We report results on COCO- Panoptic [23] and measure the mean similarity (1)).
Noticeable discrepancies exist in the between- class similarities of textual and visual features between stuff and thing classes.
Stuff classes exhibit significantly higher levels of similarity in text features than things.

---

Open
Instance Segment.
Semantic
Panoptic
Referring
Cls-agnostic
Cls-aware Part Seg.
Object Detection
Vocab.
Segment.
Segment.
Segment.
Part Seg.
SAM [24]
X
X
X
SEEM [67]
V
X
ODISE [56]
/
/
X
X
UNINEXT [58]
4
V
X
X
V
X
V
X-Decoder [66]
/
V
V
V
X
X
G-DINO [36]
/
X
X
J
X
/
PPS [5]
X
X
X
X
X
V
X
HIPIE
/
V
/
/
V
vs. prev. SOTA
+5.1 +2.0
+1.3
+0.5
+5.2 +3.2
Table 1: Our HIPIE is capahle of performing all the listed segmentation and detection tasks and achieves the sate-of-theear performance using unified framework. We present performance comparisons with SOTA methods on range of benchmark datasets: AP for instance segmentation MSCOCO [34], Ap for object detection on MSCOCO, oloU for referring segmentation on RefCOCO+ [61]. mloU for semantic segmentation on Pascal Contex[[64]. and mloUpws for part segmentation on Lascal-Panoptic-Pee performing method for each task is underlined. :: object detection can be conducted via generating bounding boxes using instance segmentation masks. 'SSeg' denotes segmentation. t: In principle, UNINEXT take arbitrary texts labels, however, original work focused on close-set performance and did not explore open-voue vecen-ocabulary inference.
[S]. The second best
This observation suggests that integrating textual features may yield more significant benefits in generating discriminative features for thing classes compared to stuff classes. Consequently, for thing classes, adopt early image-text fusion approach to fully leverage the benefits of discriminative textual features. Conversely, for stuff classes, we utilize late image-text fusion strategy to mitigate the potential negative effects introduced by non-discriminative textual features. Furthermore, the presence of discrepancies in the visual and textual features between stuff and thing classes, along with the inherent differences in their characteristics (stuff classes requiring better capture of texture and materials, while thing classes often having well-defined geometry and requiring better capture of shape information), indicates the need for decoupling the representation learning modules for producing masks for stuffs and things.
In addition to instance/semantic-lees segmentation, model is capable of open- vecc-vocabulary hierar- chical segmentation. Instead of treating part classes, like 'dog leg', as standard multi-word labels, we concatenate class names from different granularity prompts. During training, classification head using both part labels, such as 'tail', and instance labels, such as 'dog', and we explicitly contrast mask embedding with both instance-level and part-level labels. In the inference stage, we perform two separate forward passes using the same image but different prompts to gen- erate instance and part segmentation. This design choice empowers open- apen-vocabulary hierarchical segmentation, allowing us to perform part segmentation on novel part classes by randomly combining classes from various granularity, such 'giraffe' and 'leg'. even if they have never been during training. By eliminating the constraints of predefined object classes and granularity, HIPIE offers a more flexible and adaptable solution for image segmentation.
supervise the
extensively benchmark HIPIE on various popular datasets to validate its effectiveness, including MSCOCO, ADE20K, Pascal Panoptic Part, and Re|COCC//e/COCOg HIPIE achieves state-of-the- art performance across all these datasets that cover a variety of tasks and granularity.
To the best of our knowledge, HIPIE is the first hierarchical, open-vocabulary and universal image segmentation and detection model (see Table 1). By decoupling representation learning and text- image fusion mechanisms for things vs. stuff classes, HIPIE overcomes the limitations of existing approaches and achieves state-of-the-art performance on various benchmarks.
2 Related Works
Open-Vooe Vocabulary Semantic Segmentation [2, 53, 26, 16, 44, 32, 54, 55] aims to segment an
image into semantic regions indicated by text descriptions that may not have been seen during training. 7S3Net [2] combines deep visual segmentation model with an approach to generate

---

visual representations from semantic word embeddings to learn pixel-wise classifiers for novel categories. LSeg [26] uses CLIP's text encoder [43] to generate the corresponding semantic class's text embedding, which it then aligns with the pixel embeddings. OpenSeg [16] adopts a grouping strategy for pixels prior to learning sesuat-semantic alignments. By aligning each word in caption to one or few predicted masks, it can scale-up the dataset and vocabulary sizes. Group ViT' [54] is trained on a large-scale image-text dataset using contrastive losses. With text supervision alone, the model learns to group semantic regions logether. OVSegmentor [55] uses learnable group tokens to cluster image pixels, aligning them with the corresponding caption embeddings.
Veen-oocay Panoptic Segmentation (OPS) unifies semantic and instance segmentation, and aims to perform these two tasks for arbitrary categories of text-based descriptions during inference time [10, 56, 66, 67, 58]. MaskCLIP [10] first predicts class-agnostic masks using a mask proposal network. Then, refines the mask features through Relative Mask Attention interactions with the CLIP visual model and integrates the CLIP text embeddings for open-vocabulary classification. ODISE [56] unifies Stable Diffusion [47], pre-trained text-image diffusion model, with text-image discriminative models, e.g. CLIP [43], to perform open-vocabulary panoptic segmentation. X- Decoder [66] takes two types of queries as input: generic non-semantic queries that aim to decode segmentation masks for universal segmentation, and textual queries to make the decoder language- aware for various open- vocabulary vision tasks. UNINEXT [58] unifies diverse instance perception tasks into object discovery and retrieval paradigm, enabling flexible perception of open- vocabulary objects by changing the input prompts.
Referring Segmentation learns valid multimodal features between visual and linguistic modalities to segment the target object described by a given natural language expression [19, 60, 20, 22, 13, 59, 52, 35, 63]. It can be divided into two main categories: 1) Decoder-fusion based method [8, 51, 63, 35] first extracts vision features and language features, respectively, and then fuses them with a multi- modal design. 2) Encoder-fusion based method [13, 59, 30] fuses the language features into the vision features early the vision encoder.
Parts Segmentation learns to segment instances into more fine- tne-grained masks. PPP [5] established a baseline of hierarchical understanding of images by combining a scene-level panoptic segmentation model and part-level segmentation model. JPPF [21] improved this baseline by introducing joint Panoptic-Pait Part Fusion module that achieves comparable performance with significantly smaller models.
Promptable Segmentation. The Segment Anything Model (SAM) [24] is an approach for building a fully automatic promptable image segmentation model that incorporate various types of human interventions, such as texts, masks, and points, SEEM [67] proposes a unified prompting scheme that encodes user intents into prompts in joint visual-semantic space. This approach enables SEEM to generalize to unseen prompts for segmentation, achieving open- vocabvocyyulary and zero-shot capabilities. Referring segmentation can also be considered as promptable segmentation with text prompts.
Comparison with Previous Work. Table compares our HIPIE method with previous work in terms of key properties. Notably, HIPIE is the only method that supports open- ppen-vocabulary universal image segmentation and detection, enabling the object detection, instance-, semantic-, panoptic-, caarrciccl-(whokk instance, part, subpart), and eeferring-segmenttn tasks, all within single unified framework.
3 Method
We consider all relevant tasks under the unified framework of language-guides segmentation, which performs open-vocabulary segmentation and detection tasks for arbitrary text-based descriptions.
3.1 Overall Framework
The proposed HIPIE model comprises three main components, illustrated in Fig. 3:
1) Text-image feature extraction and information fusion (detailed Secs. 3.2 to 3.4): We first generate a text prompt 7 from labels referring expressions. Then, we extract image (7) and text (7) features F, En),,(T), F, Enc, using image encoder Enc, and text encoder Enc,, respectively. We then perform feature fusion and obtain fused features F F' FeatureFusion F.).

---

Image Encoder
Stuff Decoder
Image
Text Prompt
stuff queries
Panoptic Sengmet:SSggentt.
"person, sky, tree, building
Text Encoder
Text-Image Fusion
Thing Decoder
Referring Segment:
"Girl covaring her
Part Segment:
"person head, leg,
thing queries
Subpart Segment:
"person eye, neck,
Figure Diagram HIPIE for hicrarchical, universal open-vocabulary image segmentation and detection. The image and text prompts are first passed to the image and text decoder to obtain visual features Fs and text features Fz. Early fusion is then applied to merge image and text features to get F F% Two independent decoders are used for things (foreground) classes and stuff (background) classes.
2) Foreground (referred to as things) and background (referred to as stuffs) mask generation (detailed in Sec. 3.5): Each of the decoders takes in set of image features and text features and returns sets of masks, bounding boxes, and object embeddings (M, B, E). We compute the foreground and background proposals and concatenate them obtain the final proposals and masks follows:
Stuff: (M2, B2, F2)
StuffDccoder( Fu.. Ft) ThingDecoder( (T (eettreeFusiainnnttt FF, Ft)) (M, @ M2. Bj @ B2, E @ Eg)
Thing: (M1, B1, E1)
(I)
Overall: (M, B, E))
where @ denotes the concatenation operation.
Proposal and mask retrieval using text prompts (detailed Sec. 3.6): To assign class labels to these proposals, we compute the cosine similarity between object embedding E and the corresponding embedding E of class i {1, 21.....cc. For set of category names, the expression is concatenated string containing all categories. We obtain E by pooling tokens corresponding to each label from the encoded sequence Ft, For referring expressions, we taken the [CLS] token from BERT output Ex.
3.2 Text Prompts
Text prompting is a common approach used in open-vocabulary segmentation models [19, 60, 57, 58].
For open- vecabaaayy instance segmentation, panoptic segmentation, and semantic segmentation, the set of all labels concatenated into a single text prompt T, using a "." delimiter. Given an image and set of text prompts T, the model aims to classify N masks in the label space C U ("therr) }. where is the maximum number of mask proposals generated by the model.
For referring expressions, the text prompt is simply the sentence itself. The goal is to locate one mask the image corresponding to the language expression.
3.3 Image and Text Feature Extraction
We employ pretrained BERT model [6] to extract features for text prompts. Because the BERT-base model can only process input sequences up to 512 tokens, we divide longer sequences into segments of 512 tokens and encode each segment individually. The resulting features are then concatenated to obtain features of the original sequence length.
We utilize ResNet-50 [18] and Vision Transformer (ViT) [11] as base architectures for image encoding. In the case of ResNet-50, we extract multiscale features from the last three blocks and denote them as F.. For ViT, use the output features from blocks 8, 16, and 32 as the multiscale features F...
5

---

Kature
Stori
Baose
feature
Tee--
Ceooder
fectrr
Fuaion
Deccdder
Image
Fusion
1 PING F.
Decodder
& Musks
Image
Fusion
Stuff
Madoo
Stuff Decoder
feature
feature
Creooder
Sesture
b)
Various design choices for generating thing and stuff masks with arbitrary text descriptions. In version a). We use single decoder for all masks. Early fusion is applied. In version b), two independent decoders are used for things and stuff classes. Early fusion is adopted for both decoders. Version c) is identical to version b) with the only difference being that the stuff decoder not make use of early fusion.
Figure 4:
3.4 Text-Image Feature Fusion
We explored several design choices for text-image feature fusion and mask generation modules shown in Fig. 4 and Table and discovered that Fig. 4c) can give the optimal performance. We adopt bi-directional cross-attention (Bi-Xattn) to extract text-guided visual features Pre. and image-guided text features F.21- These attentive features are then integrated with the vanilla text features Fe and image features F, through residual connections, as shown below:
Fizus Fozs = Bi-Xattn Fr., Ft) (F, + Fizv: F3 Fe21)
(2)
(F%, FD
where F, and represent the visual and text-prompt features, respectively.
3.5
Thing and Stuff Mask Generation
We then generate masks and proposals for the thing and stuff classes by utilizing F and Fy that we obtained in Sec. 3.4.
Model Architecture. While architectures such as Mask2Former and MaskDINO [4, 28] can perform instance, semantic and panoptic segmentation simultaneously, models trained jointly show inferior performance compared with the same model trained for specific task (e.g. instance segmentation only). We hypothesize that this may result from the different distribution of spatial location and geometry of foreground instance masks and background semantic masks. For example, instance masks are more likely to be connected, convex shapes constrained by bounding box, whereas semantic masks may be disjoint, irregular shapes spanning across the whole image.
To address this issue, in a stark contrast to previous approaches [58, 36, 57] that use a unified decoder all both stuffs and things, we decouple the stuff and thing mask prediction using two separate decoders. For the thing decoder, we adopt Deformable DETR [65] with a mask head following the UNINEXT [58] architecture and incorporate denoising procedures proposed by DINO [62]. For the stuff decoder, we use the architecture of MaskDINO [28].
Proposal and Ground-Trun Matching Mechanisms. We make the following distinctions between the two heads. For thing decoder, we adopt SimOTA [15] to perform many-to-one matching between box proposals and ground truth when calculating the loss. We also use box-iou-based NMS to remove duplicate predictions. For the stuff decoder, we adopt one-to-one Hungarian matching [25]. Additionally, we disable the box loss for stuff masks. We the number of queries 900 for the things and 300 for the stuffs.
For both decoders, calculate the class logits as the normalized dot product between mask embeddings (M) and text embeddings (F)). We adopt Focal Loss [33] for classification outputs, L1 loss, and GloU loss [45] for box predictions, pixel-wise binary classification loss and DICE loss [49] for mask predictions. Given predictions (M1, B1, F1), (M2, B2, F22,, groundtruth labels (M', B', C") and its foreground and background subset (M', B'y, Cy) and (M,, By, C.), The final Loss is computed
Loss Functions.
Laing
(ai Cyt) + Arast Cmaste (M1, M2) + Abox Lnox (Bi, B') Lots (E2, C²) Amask Cmask (My, M + \box Loox box (By, BB)
Eauat =
(3)
C
Lihing + Lrulf
where Lrox = -.. + >Agion Lmack =
dve Agice and Let, Liocal- Note that while do not use the stuff decoder for thing prediction, still match its predictions with things and

---

uossed
pew
torso
face
nose
HIPIE
TICE
mouth
upper
Figure 5: Hierarchical segmentation pipeline. We concatenate the instance class names and part class names as lahels. During the training process, we supervise the classification head using both part labels and instance labels. During inference, perform separate forward passes using the same image but different prompts to generate instance and part segmentations. By combining part segmentation and instance segmentation of the same image, we obtain hierarchical segmentation results on the right side.
compute the class and box losses in the training. We find such auxiliary loss setup make the stuff decoder aware of the thing distribution and imporves the final performance.
3.6 gpen-Vocabulary Universal Segmentation
In closed set setting, we simply merge the output two decoders and perform the standard postpro- cessing of UNINEXT [58] and MaskDINO [28] obtain the final output.
In zero-shot open vocabulary setting, we follow ODISE [56] and combining our classification logits with text-image discriminative model, e.g. CLIP [43]. Specially, given the mask M on image T, its features and test classes Crest, we first compute the probability (E Cisst) = P(Cies] in the standard way as mentioned before. We additionally compute mask-pooled features of M from the vision encoder of CLIP as ECLP MaskPooling M, V())). Then we compute the CLIP logits EE, Cres) ( the similarity between the CLIP text features and the ECLP. Finally we combine the final prediction as
Piial Fr, Crea) X p1 (F, P2( Crea)
Where À is balancing factor. Emprically, we found such setting leads to better performance than naively relying completely on CLIP features only or close-set logits.
3.7 Hierarchical segmentation
In addition to the instance-level segmentation, we can also perform part-aware hicrarchical segmenta- tion. We concatenate the instance class names and part class names as labels. Some examples are "human ear", and "cat head". In the training process, we supervise the classification head with both part labels and instance labels. Specifically, we replace Lats with LetsPart Lelshhing in Eq. (3). We combine parts segmentation and instance segmentation of the same image to get part-aware instance segmentation. Additionally, layers of hierarchy is obtained by grouping the parts. For example, the "head" consists of ears, hair, eyes, nose, etc. Fig. 5 illustrates this process. Fig. A1 highlights the difference of our approach with other methods.
3.8
Class-aware part segmentation with SAM
We can also perform the class-aware hierarchical segmentation by combining our semantic output with class-agnostic masks produced by SAM [24]. Specifically, given semantic masks M, their class probability PM. and SAM-generated part masks S, we compute the class probability of mask S; E S with respect to class j as
Ps(S & Σ PM ns,
(S)
MrEM
Where JM O S. is the area of intersection between mask Mj and S,. We combine our semantic output with SAM because our pretraining datasets only contains object-centric masks, whereas the SA-IB dataset used by SAM contains many local segments and object parts.
7

---

UNNEXT
UNIEETT
Known Helarchy Dog
Hararchy
Grato
Figure 6: Qualitative Analysis of Open Vocabulary Hierarchal Segmentation. Because of our hicrarchal design, our model produces better-quality masks. In particular, our model can generalize to novel hicrarchics that do not exist in part segmentation datasets.
COCO
ADE20K
PAS-P
Method
Backbone
PQ Ap
Ap mloU
PQ
AP
AP
mIoU
mloUpan
MaskCLIP [10]
VITI6
15.1
6.0
23.7
X X-Decoder [66]
FocalT
52.6
41.3
62.4
18.8
9.8
25.0
X- X-Decoder
DaViT-B
56.2
45.8
66.0
21.1
11.7
27.2
SEEM [67]
FocalT
50.6
39.5
61.2
SEEM
Da DaViT-B
56.2
65.3
ODISE [56]
ViT-H+SD
55.4
46.0
46.1
65.2
22.6
14.4
15.8
29.9
JPPF [21]
E1[Net-b5
54.4
PPS [5]
RNST269
58.6
HIPIE
RN50
52.7
45.9
53.9
59.5
18.4
13.0
16.2
26.8
57.2
HIPIE
ViT-H
58.0
51.9
61.3
66.8
20.6
15.0
18.7
29.0
63.8
Table 2: Open-vocabulary panoptic segmentation (PQ). instance segmentation (AP ) semantic segmentation
(mloU), part segmentation (mloUPass) and object detection (AP*)). N/A: not applicable. -: not reported.
4 Experiments
We comprehensively evaluate HIPIE through quantitative and qualitative analyses to demonstrate its effectiveness in performing various types of open-vocabulary segmentation and detection tasks. The implementation details of HIPIE are explained in Sec. 4.1. Sec. 4.2 presents the evaluation results of HIPIE, Additionally, we conduct an ablation study of various design choices in Sec. 4.3.
4.1 Implementation Details
Model Learning Settings can be found in our appendix materials.
Evaluation Metries. Semantic Segmentation performance is evaluated using the mean Intersection- Over-Union (mloU) metric. For Part segmeniation, we report mloU which is the mean IoU for part segmentation on grouped part classes [5]. Object Detection and Instance Segmentation results are measured using the COCO-style evaluation metric mean average precision (AP) [34]. Panoptic Segmentation is evaluated using the Panoptic Quality (PQ) metric [23]. Referring Image Segmentation (RIS) [19, 60) is evaluated with overall loU (oloU).
4.2 Results
Panoptic Segmentation. We examine Panoptic Quality (PQ) performance across MSCOCO [34] for closed-set and ADE20K [64] for open-set zero shot transfer learning. Based Table 3 model is able to outperform the previous close-set state-of-the-art using a ViT-H backbone by +1 +1.8. In addition, we match the best open-set PQ results, while being able to run on more tasks and having a simpler backbone than ODISE [56]. Semantic Segmentation. The evaluation of our model's performance on various open-vocabulary semantic segmentation datasets is presented in Table 4. These datasets include: 1) A-150: This dataset comprises 150 common classes from ADE20K [64]. 2) A-847: This dataset includes all 847 classes from ADE20K [64]. 3) PC-59: It consists of 59 common classes from Pascal Context [39]. 4) PC-459: This dataset encompasses the full 459 classes of Pascal Context [39]. 5) PAS-21: The vanilla Pascal VOC dataset [12], containing 20 foreground classes and background class. These diverse datasets enable a comprehensive evaluation of our model's performance across different settings, such as varying class sizes and dataset complexities. Table provides insights into how our model performs in handling open-vocabulary semantic segmentation tasks, demonstrating

---

A-150
A-847
CTX459
Segin
Method
Data
PQ
Ap
AP
mloU
mloU
mloU
Ap
OpenSeed
C3C5,COCO
19.7
15.0
17.7
23.4
36.
X-Decoder UNINEXT
21.8
13.1
29.6
16.1
32.2
O365,COCO,
8.9
14.9
11.9
6.4
1.8
5.8
42.1
HIPIE w/o CLIP
R.SSCOCCO.ReCOOC.PACCOO
18.1
16.7
20.2
19.8
4.8
12.2
41.0
HIPIE w/ CLIP
(CLIP)
22.9
19.0
22.9
29.0
9.7
14.4
41.6
Table 3: Open- Vocabulary Universal Segmentation. We compare against other universal multi-task segmentation
models. (+) denotes pretraining dataset of representations.
Method
A-150
PC-59
PAS-21
COCO
Decoder
Fusion Fusion (things) (stuff)
PQ Apmssk oIOU
ZS3Net [2]
19.4
38.3
LSeg+ [26, 16]
18.0
46.5
55.1
Unified
45.1 42.9
67.1
HIPIE
53.6
75.7
59.5
Decoupled
50.6 43.6
67.6
1s. prev. SOTA
47.1
+10.7
+28.3
+4.4
Unified (Fig. 4a)
44.6
Group ViT [54]
10.6
25.9
50.7
21.1
Decoupled (Fig. 4b)
50.0 44,4
77.1
OpenSeg [16]
42.1
36.1
Decoupled (Fig. 4c)
51.3 44,4
77.3
MaskCLIP [10]
23.7
45.9
Table 5:
An ablation study on different decoder and text- image fusion designs, depicted in Fig. 4. We report PQ for panoptic segmentation MSCOCO, Apmis for instance segmentation on MSCOCO, oloU for refer- ring segmentation on RefCOCO's validation set. Our final choice is highlighted in gray
ODISE [56]
29.9
57.3
84.6
65.2
HIPIE
29.0
59.3
83.3
66.8
vs. prev. SOTA
-0.9
+2.0
-1.3
+1.6
Table 4: Comparison open-vocabulary semantic segmentation. Baseline results are copied from [56].
its effectiveness and versatility detecting and segmenting wide range of object categories in real-world scenarios.
Part Segmentation. We evaluate models performance report mloU pans in Table 3. We followed the standard grouping from [5]. Our model outperforms state-of-the-ane by +5.2 in this metric. We also provide qualitative comparisons with Grounding DINO + SAM in Fig. 7. Our findings reveal that the results of Grounding SAM are heavily constrained by the detection performance of Grounding DINO. As result, they unable to fully leverage the benefits of SAM in producing accurate and fine-grained part segmentation masks.
Parts dataset [5] and
Raw Image
SAM Mask
GDINO+SAM
HIPIE+SAM
Figure 7: Results merging HIPIE SAM for class-aware image segmentation on SA-IB IB dataset. Grounded- SAM (Grounding DINO + SAM) [29, 24] cannot fully leverage the benefits of SAM in producing accurate and fine-grained part segmentation masks. Our method demonstrates fewer misclassiiications and overlooked masks across the SA- IB datasct compared the Grounded-SA. approach.

---

Object Detection
Method
Backhone
COCO COCO+ COCOg
AP
APs APM APz.
Method
Backbone
oloU
oloU
oloU
Deform. DETR [63]
RN50
46.9 29.6 50.1 61.6
MAtNeet [60]
RN101
56.5
46.7
47.6
DN-DETR [27]
RN50
48.6 31.0
52.0
63.7
VLT [9]
Dark56
65.7
55.5
53.0
UNINEXT [58]
RN50
51.3 32.6 55.7
66.5
RefTR [40]
RNI01
74.3
66.8
64.7
HIPIE
RN50
53.9 37.5 58.0
68.0
UNINEXT [58]RN50
77.9
66.2
70.0
VS. prev. SOTA
+2.6 +4,9
+2.3
+1.5
UNINEXT VST-
82.2
72.5
74.7
Cas. Mask-RCNN [3]
CNeXtL
54.8
HIPTE
RN50
78.3
66.2
69.8
ViTDet-H [31]
VIT-H
58.7
HIPIE
VIT-H
82.6
73.0
75.3
UNINEXT [58]
VIT-H
58.1 40.7
62.5
73.6
vs. prev. SOTA
+0.4
+0.5
+0.6
HIPIE
VIT-H
61.3 45.8
65.7 75.9
prev. SOTA
+3.2 +5.1 +3.2 +2.3
Table 7: Comparison on the referring image segmen-
tation (RIS) task. We evaluate the model performance the validation sets RefCOCO, RefCOCO+, and RefCOCOg datasets using overall IoU (oloU) metries.
Table 6: Comparisons on the instance segmentation and object detection tasks. We evaluate model performance on the validation set MSCOCO.
Object Detection and Instance Segmentation. We evaluate model's object detection and instance segmentation capabilities following previous works [28, 67, 56]. On MSCOCO [34] and ADE20K [64] datasets, HIPIE achieves an increase of +5.1 and +0.6 A respectively. Detailed comparisons are provided in Sec. 4.2 which demonstrate state-of-the-ee results on ResNet and ViT architectures consistently across all Average Precision metries.
Referring Segmentation. Referring image segmentation (RIS) tasks are examined using the Ref- COCO, RefCOCO+, and RefCOCOg datasets. Our model outperforms all the other alternatives by an average of +0.5 in overall loU (oloU).
4.3 Ablation Study
To demonstrate the effectiveness of our design choices for text-image fusion mechanisms and representation learning modules for stuff and thing classes, we conduct ablation study (depicted in Fig. 4) and present the results in Table 5. From this study, draw several important conclusions:
1) Text-image fusion plays critical role in achieving accurate referring segmentation results. 2) The early text-image fusion approach for stuff classes negatively impacts the model's performance on panoptic segmentation. This finding validates our analysis in the introduction section, where we highlighted the challenges introduced by the high levels of confusion in stuff's 's textual features, which can adversely affect the quality of representation learning. 3) Our design choices significantly improve the performance of panoptic segmentation, instance segmentation, and referring segmentation tasks. These conclusions underscore the importance of our proposed design choices in achieving improved results across multiple segmentation tasks.
5 Conclusions
This paper presents HIPIE, an open- open-voca,,,aay, universal, and hierarchical image segmentation model that capable of performing various detection and segmentation tasks using a unified framework, instance, part, subpart),
inculding object detection, instance-, semantic-, panoptic-, hierarchical- and referring segmentation tasks. Our key insight is that we should decouple the representation learning modules and text-image fusion mechanisms for background (i.e., referred to as stuff) and foreground (i.e., referred to as things) classes. Extensive experiments demonstrate that HIPIE achieves state-of-the-a performance diverse datasets, spanning across wide range of tasks and segmentation granularity.
Acknowledgement Trevor Darrell and XuDong Wang were funded by DoD including DARPA LwLL and the Berkeley AI Research (BAIR) Commons.
10

---

References
[1] E. H. Adelson. On seeing stuff: the perception of materials by humans and machines. In Human vision and electronic imaging VI. volume 4299, pages 1-12. SPIE, 2001.
[] Bucher, T.-H. Vu, M. Cord, and Pérez. Zero-shot semantic segmentation. NeurlPS, 2019.
[3] Z. Cai and N. Vasconcelos. Cascade r-cun: high quality object detection and instance segmentation. JEEE transactions on pattern analysis and machine intelligence, 13(5):1483-144) 2019.
[4] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attentine mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290 12901199992200 2022.
[5] D. de Geus, P. Meletis, C. Lu, X. Wen, and G. Dubbelman. Part-aware panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5485-5494, 2021.
[6] Devlin, M.-W. Chang. K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810008805, 2018.
[7] N. Dhanachandra, K. Manglem, and Y. J. Chanu. Image segmentation using k-means clustering algorithm and subtractive clustering algorithm. Procedia Computer Science, 54:764-771, 2015.
[8] H. Ding, C. Liu, S. Wang. and X. Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16321 16330, 2021.
[9] H. Ding, Liu, S. Wang, and Jiang. VI: Vision-language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[10] Z. Ding. J. Wang, and Z. Tu. Open- open-vocabulary panoptic segmentation with maskclip. arXiv preprint arXiv:2208,0890 2022.
[11] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition scale. In International Conference on Learning Representations, 2020.
[12] M. Everingham, L. Van Gaol, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
[13] G. Feng, Z. Hu, L. Zhang. and Lu. Encoder fusion network with co-attention embedding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15506-15515, 2021.
[14] D. A. Forsyth and J. Ponce. Computer vision: a modern approach. prentice hall professional technical reference, 2002.
[15] Z. Ge. S. Liu, F. Wang. Z. Li, and J. Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2 107.004300 2021.
[16] G. Ghiasi, X. Gu, Cui, and T.-Y. Lin. Scaling vocabulayy image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557. Springer, 2022.
[17] K. Hc, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[18] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] R. Hu, M. Rohrbach, and T. Darrell. Segmentation from natural language expressions. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
[20] T. Hui. S. Liu, S. Huang, G. S. Yu. F. Zhang, and Han. Linguistic structure guided context modeling for referring image segmentation. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part X 16, pages 59 75. Springer, 2020.
11

---

[21] S. K. Jagadeesh, R. Schuster, and D. Stricker. Multi-task fusion for efficient panoptic-pan segmentation. arXiv preprint arXiv:2212.07671, 2022.
[22] Y. Jing, T. Kong, W. Wang. L. Wang, L. Li, and T. Tan. Locate then segment: A strong pipeline for referring image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9858-988 98689,8772222 2021.
[23] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9404-9413, 2019.
[24] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson. T. Xiao, S. Whitehead, A. C. Berg. W.-Y. Lo. et al. Segment anything. arXiv preprint (iiv:2330.00245 2023.
[25] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly,
2(1 2(1-2):8397,, 1955.
[26] B. Li, K. Q. Weinherger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022.
[27] F. Li, H. Zhang, S. Liu, Guo, I.. M. Ni, and I.. Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16619-13627, 2022.
[28] F. Li, H. Zhang, H. xu, S. Liu, L. Zhang, L. M. Ni, and H.-Y. Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation, 2022.
[29] L. H. Li", P. Zhang*, H. Zhang", J. Yang, C. Li, Zhong. L. Wang. L.. Yuan, L. Zhang, J.-N. Hwang, K.-W. Chang, and J. Gao. Grounded inguage-image pre-training. In CVPR, 2022.
[30] Li and I., Sigal. Referring transformer: one-step approach to multi-task visual grounding. Advances in neural information processing systems, 14519652-1966, 2021.
[31] Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 23-27, 2022, Proceedings, Pari IX, pages 280-296. Springer, 2022.
[32] F. Riag, B. WU, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu. Open-vocahulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:22/0.0411 2022.
[33] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 2980-2988, 2017.
[34] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitickk Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740 750. Springer, 2014.
[35] J. Liu, H. Ding, Z. Cai, Y. Zhang. R. K. Satzoda, V. Mahadevan, and R. Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. arXiv preprint arXiv:2302.07333 2023.
[36] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang. H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training open-set object detection. arXiv preprint arXiv:2303.05499 2023.
[37] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[38] Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2017.
[39] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891 8918898, 2014.
[40] L. Muchen and S. Leonid. Referring transformer: A one-step approach to multi-task visual grounding. In Thirty-Fifte Conference on Neural Information Processing Systems, 2021.
[41] V. K. Nagaraja, V. L. Morariu, and L. S. Davis. Modeling context between objects for referring expression understanding. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part IV 14, pages 792-807. Springer, 2016.
12

---

[42] R. Nock and F. Nielsen. Statistical region merging, IEEE Transactions on patlern analysis and machine intelligence, 26(11):445--1458 2004.
[43] A. Radford, J. W. Kim, C. Hallacy, A, Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI.R, 2021.
[44] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang. J. Zhou, and J. Lu. Denseclip: Language- guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082-18091, 2022.
[45] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[46] R. Rombach, Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolutin image synthesis with latent difflusion models, 2021.
[47] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 10684-10695, 2022.
[48] S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun. Objects365: large-scale, high- quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430-8439, 2019.
[49] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop. ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC. Canada, September 14, Proceedings 3. pages 200-248. Springer, 2017.
[50] R. Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
[51] Z. Wang, Lu,, Q. X. Tao, Y. Guo, M. Gong, and T. Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages I 11686- 11695, 2022.
[52] J. Wu, X. Li, X. Li, H. Ding, Y. Tong. and D. Tao. Towards robust referring image segmentation. arXiv preprint ar wxXiv:2009.0955 09554, 2022.
[53] Y. Xian, S. Choudhury. Y. He, B. Schiele. and Z. Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256-8265, 2019.
[54] Xu, De Mello, Liu, Byeon, T. Breuel, Kautz, and Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 144, 2022.
[55] Xu, J. Hou, Zhang, R. Feng, Y. Wang, Y. Qiao, and W. Xie. Learning open- apen-vocabulary semantic segmentation models from natural language supervision. arXiv preprint arXiv:2301.09121, 2023.
[56] Xu, Liu, A. Vahdat, W. Byeon, X. Wang, and S. Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. arXiv preprint arXiv:2303.04453 04803, 2023.
[57] Xu, Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai. A simple baseline for open-vocabule vocabulary semantic segmentation with pre-trained lasgunnlanguage model. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 736-753. Springer, 2022.
[58] B. Yan, Y. Jiang, J. WU, D. Wang, P. Luo, Z. Yuan, and H. Lu. Universal instance perception as object discovery and retrieval. arXiv preprint arXiv:2303.0m 06677, 2023.
[59] Z. Yang. J. Wang, Y. Tang, K. Chen, H. Zhao, and P. H. Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18155-18165, 2022.
[60] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, 2018.
13

---

[61] L. Yu, P. Poirson, S. Yang. A. C. Berg, and T. L. Berg. Modeling context in referring expressions, In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 69-85. Springer, 2016.
[62] H. Zhang, Li, S. Liu, Zhang, H. Su, J. Zhu, L. M. Ni. and H.-Y. Shum. Dino: Detr with improved denoising anchor boxes end-to-end object detection, arXiv preprint arXiv:2203.03600 2022.
[63] W. Zhao, Y. Rao, Liu, B. Liu, Zhou, and Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.00 03.02153, 2023.
[64] B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321. 2019.
[65] X. Zhu, W. Su, L. Lu, B. Li, X. Wang. and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[66] X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, Dai, H. Behl, J. Wang. Yuan, al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11220. 2022.
[67] X. Zou, J. Yang, H. Zhang. F. Ii, Li, Gao, and Y. Lee. Segment everything everywhere all arXiv preprint Xiv:2304.06678.. 2023.
14

---

Appendix
A.1 List of Datasets
semantic
instance
panoptic
grounding
part
training
# images
ADE-150
2000
Pascal VOC
1449
Pascal Context-59
5105
Pascal-Panoptaa Panootiicrtt
10103
COCO
121408
RefCOCO
19994
RefCOCO+
19992
RefCOCOg
267
Table A1: List of the dataset used. The checkmarks denote whether dataset has particular type of annotation and whether the dataset is used in the training process. * Because of data leak between Pascal PaaalllaannppPPPePeeeeee Parts other Pascal datasets, we use weights trained without Pascal- Panoptic-Paatttt those evaluations unless otherwise specified.
We report the statistics of datasets used in training and evaluation in table Table AI. Additionally, further evaluate our model on 35 object detection datasets and 25 segmentation datasets in Sec. A.4.2. In total, we benchmarked our model on around 70 datasets. These benchmarks show our model can adapt to many different scenarios and retain reasonable performance in zero-shot manner.
A.2 Hierarchical Segmentation
HIPIE
UNINEXTT
ODISE
DDAT
BERT
airplane
log
person head
n088
0ye
head
(SWPSOR)
Figure A1: Hierarchal Design HIPIE compared with other methods.
Fig. Al highlights the difference of our approach with other methods for hierarchical segmentation. We concatenate class names of different hierarchies prompts. During the training, we uniquely contrast mask embedding with both scene-level and part-level labels explicitly. Previous works such as UNINEXT and ODISE only treat these classes as normal multi-word labels. While UNINEXT allows contrasting different words individually because of the design of BERT encoder, leads to suboptimal signals. In the example above, "person head" has both positive and negative target for "person".
A.3 Experiment Setup
A.3.1 Model Learning Settings
HIPIE is first pre-trained on Objects365 [48] for 340k iterations, using a batch size of 64 and learning rate of 0.0002, and the learning rate dropped by a factor 10 after the 90th percentile of the schedule. After the pre-training stage, we finetune HIPIE COCO [34], RefCOCO, RefCOCOg. and RefCOCO+ [41, 61] jointly for 120k iterations, using batch size of 32 and a learning rate of 0.0002. For both stages, resize the original images that the shortest side is at least 800
15

---

pixels and at most 1024 pixels, while the longest side is at most 1333 pixels. For part segmentation, we train additionally tram our model jointly on Saeeal-Panottic-Pare [5] dataset and all previously and other Pascal
mentioned datasets. Because of potential data leaks between Pascal Pascal-Panoptic-PPe datasets used in the open-vocabulary segmentation evaluation, we report those numbers with weights not trained on Pascal-Panoptic-Pel Part dataset. Because of our hierarchal design, our model produces better-quality masks. particular, our model can generalize to novel hierarchies that do not exist in part segmentation datasets. In Fig. 6, we provide visualization of such results.
A.3.2 Implementation Details
For loss functions Eq. (3), we have A = 2.0, Amsk = 5.0, Abor = 5.0, Ace = 1.0, Adice 1.0, λL =1.0, Agion 0.2. For À in Eq. (4), we use À 0.2 for seen classes during the training and À = 0.45 for novel classes. In close-set evaluation, we set X = 0.0 and do not use CLIP. We also do not use CLIP for PAS-21 evaluation (whose classes are mostly covered by COCO) because find it degrades the performance. We use 800 and 1024-resolution images during the training. For evaluations, we use 1024-resolution images.
A.3.3 Training Process
Stage
Task
Dataset
Batch Size
Max Iter
Step
OD&IS
Objects365
64
340741
312346
OD&IS
COCO
32
II
91990
76658
REC&RIS
RefCOCO/g/+
32
PanoS REC&RIS PartS
COCO
32
III
RefCOCO/g/+
32
150000
100000, 100000,135000 135000
Dassal-Panoptic-Parte
32
Table A2: Training Process. Following UNINEXT [58), We first pretrain model for object detection Ob- ject365 for 340k iteration (Stage I). Then we fine-tune our model jointly on COCO for object detection, instance segmentation, referring expression comprehension (REC), and referring segmentation (RIS) for 92k iteration (Stage II.. We further jointly train our model on Panoptic Segmentation, REC, RIS, and Part Segmentation for 150k iteration (Stage III)
We train all our models NVIDIA-AI00 GPUs with batch size of 2 per GPU using AdamW [38] optimizer. We use a base learning rate of 0.0001 and a weight decay 0.01. The learning rate of the backbone is further multiplied by 0.1. Following UNINEXT [58), We first pretrain model for object detection on Object365 for 340k iteration (Stage 1). Then we tin-ttne our model jointly on COCO for object detection, instance segmentation, referring expression comprehension (REC), and referring segmentation (RIS) for 91k iteration (Stage II). We further jointly train our model on Panoptic Segmentation, REC, RIS, and Part Segmentation for 150k iteration (Stage III). In Stage I, the learning rate is dropped by a factor of 10 after 312k iterations. In stage II, the learning rate is dropped by factor of 10 after 77k iterations. In Stage III, the learning rate is dropped by a factor of 10 after 100k and 135k iterations. In all stages, we sample uniformly across datasets when there are multiple datasets. The global batch size is 64 in Stage and 32 in Stage II and III. Notably, our stage I and II is identical to the setup of UNINEXT. For ablation studies, we train stage III only and reduce the schedule 90k iterations. The learning rate schedule also scaled accordingly. The details of training recipe is shown in Table A2.
A.4
Additional Evaluations
A.4.1 Referring Expression Comprehension
In addition to Referring Segmentation reported in Table we further report results on Referring Expression Comprehension (REC), which aims to locate target object in an image at the pixel-level, given a referring expression as input. We establish new state-of-the-art performance by an average of +0.3 P@O.5 and +0.5 oloU across three datasets.
16

---

RefCOCO
RefCOCO+
RefCOCOg
Method
Backbone
oloU
P@0.5
oloU
P@00.
oloU
P.00.
MAttNet [60]
RN1O1
56.5
76.7
46.7
65.3
47.6
66.6
VLT [9]
Dark56
65.7
76.2
55.5
64.2
53.0
61.0
RefTR [40]
RN101
74.3
85.7
66.8
77.6
64.7
82.7
UNINEXT [58]
RN50
77.9
89.7
66.2
79.7
70.0
84.0
UNINEXT [58]
VIT-H
82.2
92.6
72.5
85.2
74.7
88.7
HIPIE
RN50
78.3
90.1
66.2
80.0
69.8
83.6
HIPIE
ViT-H
82.6
93.0
73.0
85.5
75.3
88.9
vs. prev. SOTA
+0.4
+0.4
+0.5
+0.6
+0.2
Table A3: Comparison on the referring expression comprehension (REC), and referring image segmentation (RIS) tasks. The evaluation is carried out on the validation sets of RefCOCO, RefCOCO+, and RefCOCOg datasets using Precision@ @0,5 and overall IoU (oloU) metrics for REC and RIS, respectively.
Median
Aorattlag Manten
Drone large
AutalMu
Amerrpann Sign
Lang Litters
BCCD
buggk/Boords hrookish Linderwaie
ChessPecces
Cetientall
Rutbels
dicc
CodiumCoooor
DromeC Prome Oentrll
EgetHann
opeallic
Handtti
Workers
Mask Weaning
Monatain Dew
Commice
Nonhh Americaa
Musle
Nissnn
Restad Pos
by breoe
Pets
Packages
VOC
Pissols
PXLox
plantie
Pothale
Roccoon
vefidiiigg Car
Shhllfish Openiinggs
Thormal
theemal Degs
And Peuple
Uno
Openhinges
aecbalie
Stremmboot
Wildtire Stooke
Method
MDETR
3.0
0.6 0.3 6.700 0.7 3.86.5 0.0 3.8 5.
30 30 0,0 0.0
8 22,80,033 12.
GLIP-T
11.4 1.6
4.3 17.1 0.1 16.01.70.0 1.7 0.57.0 0.5 0.1 1.1
0.1
2,7 0.615.3 5.9 0.0
0.3 1.6 58.351.2816001 1.6 6.2 7.4559 0.2 38.70.055.0 0.3 0.0
5.2
3
0.7
0.377.2274
7.8 68115.666.111337
33.45.527.5
IMPE
17.9 5.5
10.91662. 288 18.38.0011 27 5.75.7 0.3 1.6 6.66 1.8 I. 8.5 42.7
7.2
56266.666.2266
2.9 49.77,349.6 0.3 5330.0335 041
Table A4: We present the object detection results in the OdinW benchmark. We report mAP and mean results averaged over 35 datascts. Notably, our ResNct-50 baseline surpasses GLIP-T by +3.1. Wc use the notation HIPIE and HIPIE denote our method with ResNet-50 and ViT-H backbones, respectively.
HouscHold
Watemndon
Airplane
Parts
Chickent
Coows
Besun
Staver
Eleephand a01s
Hand
Tones
Suteenly
Phoones
Papgles
Rall
Viller
Tables
Thas
Method
Mean Medlan
X-Deewdert
52.3
22.3
13.1
42.1
2.2
8.6
44,9
7.5 66.0 79.2 33.0
11.6 75.9
42.1
7.0
$3.0
68.4 15.6 20.1 59.0 2.3 19L0 67.1 22.5 9.9 22.3 13.8
50.1
94.2
6.7
Table A5: Segmentation Result on Segin W benchmark across 25 datascts. We report mAP. We outperform
X-Decoder by large margin (+8.9)
A.4.2 Object Detection and Segmentation in the Wild
To further examine the open- apen-vocabulary capability of our model, we evaluate on the Segmentation in the Wild (SeginW) [66] consisting of 25 diverse segmentation datasets and Object Detection in the Wild (OdinW) [29] Benchmark consisting of 35 diverse detection datasets. Since OdinW benchmark contains Pascal VOC and some of the classes in SeginW benchmark are covered by Pascal- PanoppPccopttttttttt we use a version of our model that is not trained on Rascal-Panoptic- Parrts for both benchmarks for fair comparison.
We report the results in Table A5 and Table A4. Notably, our method establishes new state-of-the of SeginW benchmark by an average of +8.9 mAP across 25 datasets. We achieve comparable performance under similar settings. In particular, our ResNet-50 bascline outperforms GLIP-T by +3.1 mAP. We note that other methods such DroundingDINO [36] achieve better absolute performance by introducing more grounding data, which can be critical in datasets whose classes are not common objects. (For example, the classes of Boggle Boards are letters, the classes of UnoCards are numbers, and the classes of Sereeshttte are UI elements).
A.5 Other Ablation Studies
We provide further ablations on a few design choices in this section.
Text Encoder. We experiment with replacing the BERT text encoder in UNINEXT with pre-trained CLIP encoder. Additionally, following practices ODISE [56), we prompt each label to a sentence "a photo of <label>". " For RIS and REC tasks, the language expression remains unchanged. We report
17

---

COCO
RefCOCO
PQ
ApMask
oloU
CLIP
51.5
44.3
48.7
BERT
51.3
44.4
77.3
Table A6: Ablation Studies on choice Text Encoder. We find that while CLIP and BERT achieve similar performance on panoptic and instance segmentation. BERT performs significantly better on Referring Instance Segmentation (+-28.6 oloU).
COCO
RefCOCO oloU
PQ
APMask
w/ OTA
50.9
43.6
76.3
w/ OTA
51.3
44.4
77.3
Table A7: Ablation Studies on the SimOTA matching process. Introducing SimOTA leads to performance
improvement in all evaluation metries.
the result in Table A6. We find that while CLIP and BERT achieve similar performance panoptic and instance segmentation, BERT performs significantly better on referring instance segmentation (+28.6 oloU). We hypothesize that this may be caused by the lack of explicit language-focused training which can help achieve a better understanding of referring expression.
Followig UNINEXT [58] adopted simOTA in the matching process for "thing" classes during the training. We experiment with removing simOTA matching and use standard one-to-one matching instead. We report the result in Table A7. We find that simOTA improves the performance on both panoptic segmentation and referring instance segmentation.
A.6 Limitations
We've showcased experimental evidence supporting our method across diverse set of tasks, including open vocabulary panoptic and semantic segmentation, instance and referring segmentation, and object detection. However, it will be crucial for future work to test our methodology on video-related tasks, such as object tracking and segmentation, to draw comparisons with other universal models like UNINEXT [58]. Furthermore, it's worth considering additional pretraining of our vision encoder on newer, more complex datasets that encompass vast amount of masks and information. For instance, SA-1B [24], which includes over billion masks, would serve as an ideal training ground. Lastly, it would be advantageous to measure the change in performance when training supplementary hierarchical datasets. Such an approach will allow us to demonstrate more varied object part segmentations, thereby expanding the capabilities and versatility of our model.
A.7 Broader Impact
Our research introduces a potent approach to hierarchical and universal open vocabulary image segmentation, aiming to address the ever-increasing demand for more data and advanced model architectures. As the demand increases, practical methodologies such as universal segmentation are projected to play a vital role in constructing and training foundational models. Our model, HIPIE, shows promise for fostering progress in a multitude of fields where hierarchical data are critical, including self-driving cars, manufacturing, and medicine. However, it's imperative acknowledge potential limitations. Given that our model is trained on human annotations and feedback, it can inadvertently replicate any errors biases present in the datasets. The architecture' complexity is further enhanced when multiple models are integrated, which, in turn, compromises the explainability of the final predictions. Therefore, as with the introduction of any novel technology, it's crucial to implement safety protocols to mitigate misuse. This includes mechanisms for ensuring the accuracy of inputs and establishing procedures to comprehend the criteria the model employs for predictions. By doing so, improve the model's 's reliability and mitigate potential issues.
18

---

A.8
Qualitative Results
A.8.1 More Visualizations
We provide more visualizations of panoptic segmentation, part segmentation and referring segmenta- tion Figs. A2 and A3.
A.8.2 Combining with SAM
We integrate our model with the mask outputs generated by the ViT-H Image encoder from Segment Anything (SAM) [24]. The encoder is trained SA-1B which encompasses broad spectrum of objects and masks within each image, enabling us to enhance our segmentation output by utilizing the high-quality masks from the SAM encoder generate finer, more detailed masks.
To elaborate, in the context of panoptic segmentation, we implement a voting scheme between our pixel-wise annotations and the masks from Segment Anything (SAM), enriching these masks with our labels. For objects where our model demonstrates a strong understanding of hierarchy, such as "person" or "bird", substitute the SAM masks with ours. This approach enables to optimize hierarchical outcomes in the face of highly complex images.
Based on our observations from the figures, it's evident that Grounding DINO generates instance segmentation bounding boxes and subsequently uses SAM for the application of the segmentation masks. While this method proves effective for most datasets, SA-1B is a highly complex set featuring a vast array of whole objects, parts and subparts. Our qualitative findings suggest that the a single granularity instance segmentation model may fail to fully capture all objects/parts within an image or may incorrectly identify them. This consequently leads to SAM receiving sub-optimal bounding boxes for segmentation, resulting in fewer and less accurate masks (see third columns in Figs. A4 to A6). contrast, our methodology (see last columns in Figs. A4 to A6) integrates the SAM encoder masks with our annotations and hierarchical masks wherever feasible. This results in a significantly more fine-grate mne-grained and accurate output, proving superior in handling complex datasets such as SA-
A.8.3 Combining with Stable Diffusion
As an interesting experiment, Diffusion[46] in Fig. A7. Given a source expression and target prompt, we first use HIPIE's segmentation capability find the corresponding masks, which are then used for image inpainting. Notably, our model can uniquely achieve fine-graine grained control over object parts by providing part segmentation masks.
combined model with image generation model Stable-
19

---

Panoptic Segment
Part Segment
Subpart Segment
Referring Segment
Figure A2: More visualizations showcasing panoptic segmentation, part segmentation, subpart segmentation, and referring segmentation results on RefCOCO. It recommended to view the results color and zoom for better detail.
20

---

Minse
Panoptic Segment
Part Segment
Subpart Segment
Referring Segment
Figure A3: More visualizations showcasing panoptic segmentation, part segmentation, subpart segmentation, referring segmentation results on RefCOCO. It recommended to view the results color and zoom for better detail.
21

---

Raw Image
SAM Mask
GDINO+SAM
HIPIE+SAM
Figure A4: Additional results of merging HIPIE with SAM for hierarchical segmentation. By integrating the part masks from our model and conducting vote among SAM's panoptic masks, we generate finely detailed mask outputs. Our method demonstrates fewer misclasiffications dataset compared to the Grounding DINO SAM approach. Furthermore, our technique excels in differentiating between intra-class objects and identifying distinct object parts.
overlooked masks across the SA- IB
22

---

Raw Image
SAM Mask
GDINO+
HIPIE+SAM
Figure A5: Additional results of merging HIPIE with SAM for hierarchical segmentation. By integrating the part masks from our model and conducting vote among SAM's panoptic masks, we generate finely detailed mask outputs. Our method demonstrates fewer misclasiffications and overlooked masks across the SA- IB dataset compared to the Grounding DINO SAM approach. Furthermore, our technique excels in differentiating between intra-class objects and identifying distinct object parts.
23

---

Raw Image
SAM Mask
GDINO+SAM
HIPIE+SAM
Figure A6: Additional results of merging HIPIE with SAM for hierarchical segmentation. By integrating the part masks from our model and conducting vote among SAM's panoptic masks, we generate finely detailed mask outputs. Our method demonstrates fewer misclasiifications and overlooked masks across the SA- 1B dataset compared to the Grounding DINO SAM approach. Furthermore, our technique excels in differentiating between intra-class objects and identifying distinct object parts.
24

---

Segmentation
Generation
Prompt
Change the dog to black cat, high resolution, sitting park bench
Change the dog's head to the head of cat
Change the woman Spiderman wearing his suit
Change woman's hair to
Blonde Curly Hair
Figure Results of combining HIPIE with Stable Diffusson Image inpainting. We leverage our segmenta- tion model generate masks for the redrawing process. Our model can uniquely achieve fine-grained control by providing part segmentation masks.
25


Reviews:
[{"review_id": "93O0HWspTD", "reviewer": "Reviewer_YNZH", "summary": "The paper propose to disentangle the representation learning and decoding for things and stuff and unify multiple segmentation tasks with different granularity (whole, part, subpart) and text formulation (reference text or category only). Extensive experiments are carried out to validate its effectiveness.", "questions": "Generally, I appreciate the setting of this work, but still have some concerns as detailed in the weakness part. If the author can address my concern, I am willing to upgrade my rating.\n\nBesides, I hope the author can have a detailed introduction about how to do the hierarchical segmentation in 3.7, you need to do the segmentation multiple times or just once? If the text query has both whole, part and subpart categories, how will the model do the inference.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed to unifying different segmentation tasks and benchmark them, which is a promissing direction for future researches.\n\n2. The decoupling of things and stuffs in representation learning are intuitive.\n\n3. The experimental results are extensive and promissing.", "weaknesses": "1. As the model only uses fully supervised data for supervision without large-scale image-text pretraining, whether its open-vocabulary capability is enough to be claimed as a open-vocabulary model? I am looking forward to see more open-vocabulary results that a quite different with exisitng training vocabulary with and without the assistance of CLIP for inference.\n\n2.  It seems that the architecture designs (decoupling thing and stuff) of this work do not feasible to the whole/part/subpart seg. If previous methods such as UNINEXT, X-Decoder, SEEM  are also trained with part segmentation dataset (e.g., Pascal-Panoptic-Part). Can it already provide a solid results for such unified segmentation? Better with some experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your invaluable insights and thoughtful comments. In the following sections, we address the questions you have raised:\n\n**[W1] Open-vocabulary results with and without the assistance of CLIP for inference**\n\nNice question! We sincerely appreciate the reviewer for highlighting the absence of extensive large-scale image-text pre-training. This gap indeed serves as one of the motivations behind our integration of CLIP assistance (it is also a common practice). \n\nTo answer your question in full, we argue that the extensive labels in detection dataset and language expression in RefCOCO/+/g can also help the model align the representations of image and texts. This alignment, in turn, contributes to our model's capability to achieve open vocabulary segmentation without using CLIP. \n\nWe provide additional results in the table below. In comparison to other universal models, our model, when integrated with CLIP, achieves the state-of-the-art performance in ADE-full benchmark with 847 classes. Moreover, the performance exhibited by HIPIE, even in the absence of CLIP, surpasses the UNINEXT baseline by a significant margin. This outcome underscores our model's commendable open vocabulary capability, even without reliance on CLIP assistance.\n\n|Method|Venue|Dataset|A-150 (PQ)|A-150 (APmask)|A-150 (APbox)|A-150 (mIoU)|A-847 (mIoU)|CTX-459 (mIoU)|SeginW (APmask)|\n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |\n|OpenSeed|ICCV2023|O365,COCO|19.7|15.0|17.7|23.4|-|-|36.1|\n|X-Decoder|CVPR2023|COCO,CC3M,SBU-C,VG,COCO-Caption,(Florence)|21.8|13.1|-|**29.6**|9.2|**16.1**|32.2|\n|UNINEXT|CVPR2023|O365,COCO,RefCOCO|8.9|14.9|11.9|6.4|1.8|5.8|42.1|\n|HIPIE w/o CLIP (ours)|-|O365,COCO,RefCOCO|18.1|16.7|20.2|19.8|4.8|12.2|41.0|\n|HIPIE w/ CLIP (ours)|-|O365,COCO,RefCOCO, (CLIP) |**22.9**|**19.0**|**22.9**|29.0|**9.7**|14.4|**41.6**|\n\n**[W2 and Questions] How to do hierarchical segmentation? If previous methods such as UNINEXT, X-Decoder, SEEM are also trained with part segmentation dataset (e.g., Pascal-Panoptic-Part). Can it already provide solid results for such unified segmentation?** \n\nWe are sorry for the confusion caused by insufficient architectural description. We have indeed incorporated distinct designs specifically tailored for hierarchical segmentation. In our efforts to elucidate the pivotal differences from previous methods and from naively training the model on part-segmentation datasets, we have included illustrative diagrams in the rebuttal PDF.\n\nSpecifically, we concatenate class names from various hierarchical levels and contrast a mask embedding with these labels within the training loss. To illustrate, consider the example of \"person head\", we establish positive targets for both \"person\" and \"head\" individually, while designating negative targets for all other class names. This approach starkly contrasts with the outcomes of naively applying alternative methods, where \"person head\" might unintentionally garner negative targets from classes like \"person body\" or \"person eye.\"\nInstead of treating each class name as an ordinary multi-word class label, our design uniquely captures the hierarchical nature of the underlying semantics. At inference, we run the same image once for each level of hierarchy and combine the final outputs. \n\nIn Figure R1, we visually articulate the design disparities when compared to methods like UNINEXT and ODISE. \n\nIn Figure R2, we show that our design benefits open-vocabulary settings and allows zero-shot inference for object parts on novel concepts. \n\nAdditionally, we have conducted performance comparisons between UNINEXT and HIPIE, both trained within the hierarchical segmentation settings. Our evaluation is conducted on the val sets of two datasets: COCO for Panoptic Segmentation and PAS-P for Part-Segmentation.\n\n|Method|Train Data|COCO (PQ)|COCO(APmask)|COCO (APbox)|COCO (mIoU)|PAS-P (mioUPartS)|\n|-|-|-|-|-|-|-|\n|UNINEXT (H)|O365,COCO,RefCOCO,PAS-P|37.3|60.1|49.9|21.3|52.0|\n|HIPIE (H)|O365,COCO,RefCOCO,PAS-P|**58.0**|**61.3**|**51.9**|**66.8**|**63.8**|\n\n*Hope our explanation and experiments are able to address your inquiries. Please don't hesitate to reply if you have any further concerns. We will integrate all your valuable suggestions into our revision, and open-source the code! Thank you!*"}, {"review_id": "0Qwz9KTxYe", "reviewer": "Reviewer_EcWi", "summary": "The paper proposes a unified method for open-vocabulary universal image segmentation and detection methods. A text-image fusion module takes both the image features and text features and then sends the fused results to the decoder. Several designed choices are presented and compared here. The model utilizes the dataset of Objects365, COCO, RefCOCO, RefCOCOg and RefCOCO+ for training and then test on different benchmarks.", "questions": "Table 3 seems confusing, why do the authors split the table into two parts?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The overall method is clear and easy to understand.\n2. The motivation to unify all the open-vocabulary segmentation and detection tasks is good.", "weaknesses": "1. The model is first pretrained on Object365 which has over 600K well-labeled images and then finetuned on COCO, RefCOCO, RefCOCOg and RefCOCO+. Previous open-vocabulary image segmentation methods like [1-3] only trains on COCO dataset. I don't think the comparison is fair as the model has seen much more well-labeled datasets.\n2. Since the hierarchical segmentation only requires the change of text prompts, I think all previous work can also perform on this task. Maybe the authors need to tone down the hierarchical thing and add some comparison with previous work.\n3. The paper is titled open-vocabulary, however, lots of results in the paper are actually not such as the COCO results in Table 2, 5, 6. It's ok to report the results, but the authors need to report the open-vocabulary results which should be the main focus.\n4. Some important comparisons are missing [4, 5] on open-vocabulary semantic segmentation.\n\n[1] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels.\\\n[2] Open-vocabulary panoptic segmentation with maskclip.\\\n[3] Open-vocabulary panoptic segmentation with text-to-image diffusion models.\\\n[4] Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.\\\n[5] Side Adapter Network for Open-Vocabulary Semantic Segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful inquiries, and we will provide detailed responses to each of them below:\n\n**[W1] Concern about HIPIE's pre-training on Object365, which is unfair.**\n\nWith regard to the use of large datasets, we'd like to highlight that our research is centered on universal models capable of addressing multiple tasks within a unified framework. As a result, we've trained our model on various datasets that collectively support these target tasks. This approach is aligned with recent endeavors such as OpenSeed (ICCV2023), GroundingDINO (arXiv2023), and X-Decoder (CVPR2023), which actually have used datasets of considerably larger scales than ours. \n\nIn the table provided below, we present a comprehensive comparison against con-current works that have been trained under comparable settings. Importantly, our work shows a remarkable performance superiority and obtains significant outperformance of the UNINEXT baseline—a model that also holds universal capabilities.\n\n|Method|Venue|Dataset|A-150 (PQ)|A-150 (APmask)|A-150 (APbox)|A-150 (mIoU)|A-847 (mIoU)|CTX-459 (mIoU)|SeginW (APmask)|\n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |\n|OpenSeed|ICCV2023|O365,COCO|19.7|15.0|17.7|23.4|-|-|36.1|\n|X-Decoder|CVPR2023|COCO,CC3M,SBU-C,VG,COCO-Caption,(Florence)|21.8|13.1|-|**29.6**|9.2|**16.1**|32.2|\n|UNINEXT|CVPR2023| O365,COCO,RefCOCO|8.9|14.9|11.9|6.4|1.8|5.8|42.1|\n|HIPIE (ours)|-| O365,COCO,RefCOCO|**22.9**|**19.0**|**22.9**|29.0|**9.7**|14.4|**41.6**|\n\nIn contrast to prior open-vocabulary segmentation methods, our HIPIE harnesses a unique advantage through the universality inherent in our model, which empowers us to effectively exploit both segmentation and detection datasets such as Object365. Prior works such as [3] and X-Decoder cannot use bounding-boxes-only datasets because of their decoder design. Other works such as [1,2,4,5] focus on semantic segmentation and do not have instance awareness.\n\n**[W2] Maybe the authors need to tone down the hierarchical thing and add some comparison with previous work.**\n\nSorry for the confusion! We have indeed integrated unique designs tailored to hierarchical segmentation. In our efforts to elucidate the pivotal differences from previous methods and from naively training the model on part-segmentation datasets, we have included illustrative diagrams in the rebuttal PDF. Specifically, we concatenate class names from various hierarchical levels and contrast a mask embedding with these labels within the training loss. To illustrate, consider the example of \"person head\", we establish positive targets for both \"person\" and \"head\" individually, while designating negative targets for all other class names. This approach starkly contrasts with the outcomes of naively applying alternative methods, where \"person head\" might unintentionally garner negative targets from classes like \"person body\" or \"person eye.\" Instead of treating each class name as an ordinary multi-word class label, our design uniquely captures the hierarchical nature of the underlying semantics. At inference, we run the same image once for each level of hierarchy and combine the final outputs.\n\n**In Figure R1**, we visually articulate the design disparities when compared to methods like UNINEXT and ODISE.\n\n**In Figure R2**, we show that our design benefits open-vocabulary settings and allows zero-shot inference for object parts on novel concepts.\n\nAdditionally, we have conducted performance comparisons between UNINEXT and HIPIE, both trained within the hierarchical segmentation settings. Our evaluation is conducted on the val sets of two datasets: COCO for Panoptic Segmentation and PAS-P for Part-Segmentation.\n\n|Method|Train Data|COCO (PQ)|COCO(APmask)|COCO (APbox)|COCO (mIoU)|PAS-P (mioUPartS)|\n|-|-|-|-|-|-|-|\n|UNINEXT (H)|O365,COCO,RefCOCO,PAS-P|37.3|60.1|49.9|21.3|52.0|\n|HIPIE (H)|O365,COCO,RefCOCO,PAS-P|**58.0**|**61.3**|**51.9**|**66.8**|**63.8**|\n\n**[W3] Open-vocabulary results should be the main focus**\n\nWe thank the reviewer for pointing out the importance of open-vocabulary performance. The closed-set results shown in Table 2, 5, and 6 primarily underscore the universality of our model, a core contribution we emphasize. This universal capability is particularly noteworthy, considering that many prior works like X-Decoder and ODISE lack the capacity to perform detection and referring expression comprehension due to their decoder designs.\n\nIn addition to the open-vocabulary results reported in the main paper, we also provide more results on Object Detection in the Wild (ODinW) and Segmentation in the Wild (SeginW) in our appendix and we obtain significantly better results. Furthermore, we provide more open-vocabulary results in the first Table (rebuttal PDF) with ADE-full and Pascal-Context-459 featuring many novel classes. Our model achieves competitive performance in these datasets as well.\n\n**[W4] Comparisons with [4, 5] are missing on open-vocabulary semantic segmentation.**\n\nWe will add it in the revision! We can outperform [4] (+0.7 mIoU on ADE-full and + 2.1 mIoU on CTX-459) and we do not surpass [5]. However, it's important to highlight that these models are designed specifically for semantic segmentation, whereas our model's capabilities encompass detection, instance segment, referring segment, and part segmentation as well. Also, these models rely on the COCO-Stuff dataset, which has more stuff classes compared to the COCO-Panoptic dataset that we employ, while our current setting is more favorable for object detection and instance segmentation. We intend to explore the implications of the COCO-Stuff dataset in the revision.\n\n**[Questions]** The first part of Table 3 presents a comparison with methods using RN50 as their backbone, while the second part lists models utilizing larger backbones. We will fix it in the revision.\n\n*Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!*"}, {"review_id": "G4MVvMZzt8", "reviewer": "Reviewer_4C24", "summary": "This paper presents HIPIE, an open vocabulary image segmentation model that produces segmentation from text prompts. The authors propose to decouple the segmentation of “thing” and “stuff” due to the differences in their semantic and geometric properties. By training on an additional part-level dataset, HIPIE can also perform part-level segmentation. The authors perform experiments on several open-vocabulary segmentation and referring image segmentation datasets and achieve better performances than previous state-of-the-art methods.", "questions": "Table 3 – what is the difference between the top part and the bottom part?", "limitations": "The authors present a section for limitations (Section A5) in the supplementary material. It points to several future work directions rather than limitations of the proposed model. One potential limitation is that the part-level segmentation might not generalize to vocabulary beyond the training set.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The decoupling of thing and stuff decoding makes sense because of the different feature distributions between the thing classes and the stuff classes. The authors have also experimentally verified that decoupling helps in quantitative measures.\n\n- The proposed method has strong performance on several popular datasets including COCO panoptic segmentation, ADE20K, the referring COCO dataset, etc. \n\n- The authors show that the proposed method can also work on part-level segmentation after being trained on part segmentation datasets. They show that the proposed method works better than Grounding DINO on at least one example in part segmentation which is an interesting application.\n", "weaknesses": "- The paper claims a hierarchical representation which I find weak. There is only a small paragraph (Section 3.7) about hierarchical segmentation, and it is about part-level segmentation. From the text, it seems like the hierarchy is embedded in the text prompt (e.g., a head consists of ears, hair, etc.) – which any model that works on a text prompt is already capable of incorporating. The proposed model itself is not hierarchical. One thing that the proposed model does while prior work doesn’t is that it trains on part segmentation dataset (Pascal-Panoptic-Parts). However, it seems that the authors have only tested part segmentation performance on the same dataset, hence not demonstrating the “open vocabulary” capability. Overall, I find the claim of “hierarchical model” as a contribution confusing.\n\n- Most of the performance improvements seem to come from decoupling. It appears to me that decoupling leads to extra model parameters and run-time because there are two decoders to run for each input image. There are no comparisons with recent models in these two regards.\n\n- It is unclear why “feature fusion” should not be performed for the “stuff” branch.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your invaluable insights and thoughtful comments. In the following sections, we address the questions you have raised:\n\n**[W1] How does the hierarchical segmentation process occur in the proposed model? Is the model inherently hierarchical in its architecture? How to demonstrate the “open vocabulary” capability for part-segmentation?**\n\nNice questions! Sorry for the confusion! We have indeed integrated unique designs tailored to hierarchical segmentation. In our efforts to elucidate the pivotal differences from previous methods and from naively training the model on part-segmentation datasets, we have included illustrative diagrams in the rebuttal PDF.\n\nSpecifically, we concatenate class names from various hierarchical levels and contrast a mask embedding with these labels within the training loss. To illustrate, consider the example of \"person head\", we establish positive targets for both \"person\" and \"head\" individually, while designating negative targets for all other class names. This approach starkly contrasts with the outcomes of naively applying alternative methods, where \"person head\" might unintentionally garner negative targets from classes like \"person body\" or \"person eye.\"\nInstead of treating each class name as an ordinary multi-word class label, our design uniquely captures the hierarchical nature of the underlying semantics. At inference, we run the same image once for each level of hierarchy and combine the final outputs.\n\n***In Figure R1***, we visually articulate the design disparities when compared to methods like UNINEXT and ODISE.\n\n***In Figure R2***, we show that our design benefits open-vocabulary settings and allows zero-shot inference for object parts on novel concepts. \n\nAdditionally, we have conducted performance comparisons between UNINEXT and HIPIE, both trained within the hierarchical segmentation settings. Our evaluation is conducted on the val sets of two datasets: COCO for Panoptic Segmentation and PAS-P for Part-Segmentation.\n\n|Method|Train Data|COCO (PQ)|COCO(APmask)|COCO (APbox)|COCO (mIoU)|PAS-P (mioUPartS)|\n|-|-|-|-|-|-|-|\n|UNINEXT (H)|O365,COCO,RefCOCO,PAS-P|37.3|60.1|49.9|21.3|52.0|\n|HIPIE (H)|O365,COCO,RefCOCO,PAS-P|**58.0**|**61.3**|**51.9**|**66.8**|**63.8**|\n\nIn the context of open-vocabulary part-segmentation, we acknowledge that a quantitative assessment of the open vocabulary capability is hindered by the limited availability of part-segmentation datasets, where most datasets have similar classes. However, we provide qualitative analysis in Figure R2, where we showed that our design benefits open-vocabulary settings and allows zero-shot inference of parts on novel objects. We are committed to further advancing our research in open-vocabulary part-segmentation. In future endeavors, we plan to label a new dataset and release it for the evaluation of open-vocabulary part-segmentation. \n\n**[W2] It appears to me that decoupling leads to extra model parameters and run-time because there are two decoders to run for each input image. There are no comparisons with recent models in these two regards.**\n\nAnother nice question! Compared with UNINEXT, we introduce 30M more parameters, or 4% increase in total parameters (805M vs 775M). In terms of inference speed, our model leads to a small performance loss on A100 (1.31s vs 1.42s per iteration). However, given the observed performance gain and new task capabilities, we believe such cost is justifiable.\n\n**[W3] It is unclear why “feature fusion” should not be performed for the “stuff” branch**\n\nRegarding the decision to refrain from \"feature fusion\" within the \"stuff\" branch, we would like to reference the analysis detailed in Section 1 (Lines 35-64) and Figure 2 of our paper. Specifically, in Figure 2, we observed that: 1) Noticeable discrepancies exist in the between class similarities of textual and visual features between stuff and thing classes. 2) Stuff classes exhibit significantly higher levels of similarity in text features than things. This observation suggests that integrating textual features may yield more significant benefits in generating discriminative features for thing classes compared to stuff classes. Consequently, for thing classes, we adopt an early image-text fusion approach to fully leverage the benefits of discriminative textual features.\n\nFurthermore, we have conducted empirical validation on MSCOCO, as presented in the table (copied from Table 4 in our paper), which underscores the superiority of our design in comparison to alternative design choices. \n\n| Method | PQ | APmask | mIoU |\n| ------ | --------- | --------- | --------- |\n| Baseline - Fig. 4a | 44.6 | 42.5 | 66.8 |\n| Decoupled (Fusion: Stuff + Things) - Fig. 4b | 50.0 | **44.4** | 77.1 |\n| Decoupled (Fusion: Things) - Fig. 4c   | **51.3** | **44.4** | **77.3** |\n\n*Hope our explanation and experiments are able to address your inquiries. Please don't hesitate to reply if you have any further concerns. We will integrate all your valuable suggestions into our revision, and open-source the code! Thank you!*"}, {"review_id": "HMmg9Agnmw", "reviewer": "Reviewer_7gze", "summary": "In this paper, the authors target  open-vocabulary setting and propose a universal framework for open-vocabulary semantic/instance/panoptic segmentation.  The whole framework is DETR-like. And to deal with the discrepancies between the thing and stuff classes, the authors ultilize independent decoders for thing and stuff classes. The authors benchmark their method on various datasets and achieve remarkable performance.", "questions": "See weakness.", "limitations": "NaN", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The idea is simple and easy to understand.\n2) The results seem promising.", "weaknesses": "1) The idea seems incremental. Compared to UNITEXT, little new knowledge is introduced. The authors try to claim that their paper focuses on the open-vocabulary setting, while UNITEXT does not. However, they do not make it clear what prevents the extension of UNITEXT to the open-vocabulary setting, and their particular design for such setting. In fact, as noted on UNITEXT's github project homepage, the approach extends easily to open-vocabulary setting and achieves better performance than the proposed approach on the Seginw benchmark.\n\n2) The comparison with other open-vocabulary semantic/panoptic segmentation methods is unfair. Since the authors used many datasets with much overlap with ADE20k-150 or Pascal Context 59 for training, but not in the previous methods, I suspect that the performance gain achieved by the proposed method comes from the training dataset. The authors should evaluate their method on a harder dataset (e.g., ADE20K-full) and perform an analysis of the overlap between their training dataset and the test data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your invaluable insights and thoughtful comments. In the following sections, we address the questions you have raised:\n\n**[W1] New knowledge introduced to UNINEXT and performance comparison with UNINEXT on open-vocabulary and part-segmentation benchmarks** \\\nWe thank the reviewer for taking note of the open-vocabulary capabilities of UNINEXT, which was released after our paper submission deadline. However, it's crucial to highlight that despite UNINEXT's strong performance in instance segmentation tasks, such as SeginW, its architecture lacks the capacity to effectively execute panoptic and semantic segmentation involving stuff/background classes. \nAs a result, it's incapable of tackling certain common open-vocabulary benchmarks like ADE-150 panoptic segmentation and ADE-Full semantic segmentation. In Table 4 of our main paper, we highlight the substantial performance gains achieved in closed-set scenarios.\nMoreover, in the table provided below (please refer to Table R1 for comprehensive results), we present the evaluation outcomes of UNINEXT when applied naively to ADE-150, ADE-Full and CTX-459 and SeginW benchmarks. Our approach surpasses UNINEXT in both AP, which assesses instance capability, PQ, which evaluates panoptic capability, and mIoU, which evaluates the semantic segmentation performance. \n|Method| Train Data | A-150 (PQ) | A-150 (APmask) | A-150 (APbox) | A-150 (mIoU) | A-847 (mIoU) | CTX-459 (mIoU) | SeginW (APmask) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| UNINEXT (H) | O365,COCO, RefCOCO | 8.9 | 14.9 | 11.9 | 6.4 | 1.8 | 5.8 | **42.1** | \n| HIPIE (H) | O365,COCO, RefCOCO | **22.9** | **19.0** | **22.9** | **29.0** | **9.7** | **14.4** | 41.6 |\n\nAdditionally, we have conducted performance comparisons between UNINEXT and HIPIE, both trained within the hierarchical segmentation settings. Our evaluation is conducted on the validation sets of two datasets: COCO for Panoptic Segmentation and PAS-P for part-segmentation. \n| Method | Train Data | COCO (PQ) | COCO (APmask) | COCO (APbox) | COCO (mIoU) | PAS-P (mioUPartS) |\n| --- | --- | --- | --- | --- | --- | --- |\n| UNINEXT (H) | O365,COCO, RefCOCO, PAS-P | 37.3 | 60.1 | 49.9 | 21.3 | 52.0 |\n| HIPIE (H) | O365,COCO, RefCOCO, PAS-P | **58.0** | **61.3** | **51.9** | **66.8** | **63.8** |\n\nIn summary, we achieve competitive, if not superior, performance to UNINEXT in instance segmentation, while significantly enhancing its performance in part-segmentation, open-vocabulary panoptic and semantic segmentation—a testament to the strides we've made in advancing the model's capabilities.\n\nWe sincerely appreciate your valuable suggestions and feedback. We will incorporate all of these results into our revision.\n\n**[W2] Is there any overlap between the training dataset used and ADE-150 or Pascal Context? Could you evaluate your approach on more challenging datasets, like ADE20K-full, and provide an analysis of data overlap between your training dataset and the test data?**  \\\nWe genuinely appreciate your insightful question, which often goes overlooked in prior research. To the best of our knowledge, we are not aware of any substantial data overlap between the datasets we employ for training and ADE-150, as well as Pascal Context. It's noteworthy that these datasets are commonly utilized in a multitude of earlier works, including OpenSeed.\n\nIn direct response to your question, we have included performance metrics on more rigorous datasets like ADE-full and Pascal-Context-459 in the Table below (please also see Table R1). Our approach exhibits superior performance compared to previous works that were trained under similar settings.\n\n|Method|Venue|Dataset|A-150 (PQ)|A-150 (APmask)|A-150 (APbox)|A-150 (mIoU)|A-847 (mIoU)|CTX-459 (mIoU)|SeginW (APmask)|\n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |\n|OpenSeed|ICCV2023|O365,COCO|19.7|15.0|17.7|23.4|-|-|36.1|\n|X-Decoder|CVPR2023|COCO,CC3M,SBU-C,VG,COCO-Caption,(Florence)|21.8|13.1|-|**29.6**|9.2|**16.1**|32.2|\n|UNINEXT|CVPR2023|O365,COCO,RefCOCO|8.9|14.9|11.9|6.4|1.8|5.8|**42.1**|\n|HIPIE (ours)|-|O365,COCO,RefCOCO|**22.9**|**19.0**|**22.9**|29.0|**9.7**|14.4|41.6|\n\nFurthermore, our model's universality empowers it to effectively harness detection datasets such as Object365—an advantage unique to our architecture. This sets us apart from models like X-Decoder, which are constrained due to their decoder design, preventing the use of bounding-box-only datasets. Similarly, other methods like [1,2,4,5] exclusively concentrate on semantic segmentation and lack instance-awareness. We argue that our universality is a strength but weakness because training on multiple tasks benefits each other.\n\n*Hope our explanation and experiments are able to address your inquiries. Please don't hesitate to reply if you have any further concerns. We will integrate all your valuable suggestions into our revision, and open-source the code! Thank you!*"}]

Author rebuttal:
We extend our gratitude to the reviewers for their valuable feedback. Their insights have significantly enriched our work.
We are heartened by YNZH's recognition of our paper, where YNZH highlights the significance of our approach in *"unifying different segmentation tasks and benchmark them, which is a promising direction for future researches"*.
We appreciate EcWi's endorsement that *"the motivation to unify all the open-vocabulary segmentation and detection tasks is good"*. 
We are pleased to acknowledge 4C24 and 7gze's observations that *"the proposed method has strong performance on several popular datasets"* and *"the results seem promising"*. Furthermore, we deeply appreciate 4C24's comment that *"the decoupling of thing and stuff decoding makes sense because of the different feature distributions between the thing classes and the stuff classes"*. 
We will integrate all valuable suggestions into our revision, and open-source the code. 

In this section, we commence by tackling the concerns that have been collectively raised. These shared concerns correspond to the three keywords in the title:

**[Hierarchical] What is the architecture design catering specifically to hierarchical segmentation and how does it compare with naively running previous methods on part segmentation dataset?** 

Sorry for the confusion! We have indeed integrated unique designs tailored to hierarchical segmentation. In our efforts to elucidate the pivotal differences from previous methods and from naively training the model on part-segmentation datasets, we have included illustrative diagrams in the rebuttal PDF. 

Specifically, we concatenate class names from various hierarchical levels and contrast a mask embedding with these labels within the training loss. To illustrate, consider the example of "person head", we establish positive targets for both "person" and "head" individually, while designating negative targets for all other class names. This approach starkly contrasts with the outcomes of naively applying alternative methods, where "person head" might unintentionally garner negative targets from classes like "person body" or "person eye."
Instead of treating each class name as an ordinary multi-word class label, our design uniquely captures the hierarchical nature of the underlying semantics. At inference, we run the same image once for each level of hierarchy and combine the final outputs. 

***In Figure R1***, we visually articulate the design disparities when compared to methods like UNINEXT and ODISE. 

***In Figure R2***, we show that our design benefits open-vocabulary settings and allows zero-shot inference for object parts on novel concepts. 

***In Table R2***, we also empirically evaluated the results, which affirm our model's superior performance compared to the UNINEXT baseline trained on part datasets.

**[Universal] What novel insights or knowledge does your approach introduce when contrasted with UNINEXT and other preceding universal models?** 

In this paper we consider the scope of all interesting scene understanding tasks consisting of Object Detection, Instance Segmentation, Part-segmentation, Semantic Segmentation, Panoptic Segmentation, Referring Segmentation and Referring Expression Comprehension. While there are many existing works approaching this objective, we are the first model capable of performing all these tasks. 

In contrast to earlier methods, it's important to note that UNINEXT lacks the capacity for panoptic and semantic segmentation and exhibits suboptimal performance when directly applied to such tasks. X-Decoder's limitations arise from its decoder not being optimized for bounding box-based learning, rendering it unable to perform object detection. Similarly, ODISE faces constraints as it cannot conduct referring segmentation and part segmentation tasks, while also demonstrating markedly inferior results in object detection and instance segmentation. A summary of these limitations is provided in Table 1 of the main paper.

**[Open Vocabulary] More open vocabulary results are required to demonstrate the effectiveness of the method. Additionally using dataset such as O365 seems unfair when compared with previous works.** 

In addition to the comprehensive evaluation results across **40** datasets presented within both the main paper and appendix materials, we have provided additional open-vocabulary results in Table R1 (see rebuttal PDF). In direct comparison to other universal models trained within similar settings, our model achieves the state-of-the-art performance in ADE-full benchmark with over 800 classes. 

Compared with methods featuring segmentation-specific decoders (e.g. ODISE and X-Decoder), our model exhibits a distinctive capability – the capacity to leverage detection datasets such as Object365 for enhanced object localization. We consider this to be a strength rather than a limitation, as annotating bounding boxes is significantly simpler compared to annotating segmentation masks. 

In contrast to prior methods primarily centered on open-vocabulary semantic segmentation, such as OVSeg or SAN, our model showcases a distinct capability by performing a considerably broader range of tasks with only a modest increase in the overall training dataset. All these models (OVSeg and SAN) used large scale image-text dataset such as LAION,CLIP, Florence. (Additionally, it's important to acknowledge that such a comparison might exhibit a bias towards semantic segmentation methods, mainly due to their narrower focus and common utilization of COCO-Stuff dataset, which contains a greater number of stuff classes compared to COCO-Panoptic.) 

*We believe that the explanations provided above sufficiently address the primary concerns shared by the reviewers. We kindly invite you to review the individual comments for each reviewer, where you will find our responses to reviewers' specific inquiries. Your consideration is greatly appreciated. Thank you!*


Meta-review: This is a system paper for hierarchical open-vocabulary universal image segmentation. The reviews are divergent, even after the discussion stage post rebuttal, with 1 weak accept, 2 borderline accepts, and 1 borderline reject.

On the positive side: 1). It's a system paper with carefully designed/adopted individual modules for image feature and text embeddings, hierarchical design, separation of the stuff and things decoding. 2). Decodings for the stuff and things have been separate, which is somewhat different from the previous approaches. 3). The hierarchical objects with the parts have been addressed.

On the negative side: 1). Even though there is a big claim about hierarchical segmentation in the title, the description about the algorithm design is nearly none in the original submission, although more explanation has been provided in the rebuttal. 2). It's a system term and there didn't appear to have existed anything that is particularly novel. 3). After all, the algorithm is still under a supervised setting, making it's open-vocabulary capability under concerns (a common point shared among reviewers). It requires to bring in the CLIP modules to enhance its open-vocabulary capability. 4). The benchmark results against the existing approaches on the open-vocabulary setting are considered somewhat unfair.

Authors have provided several tables with new results to answer the questions raised by the reviewers. Overall, these results are appreciated by the reviewers with some initial concerns resolved or partially resolved. However, concerns regarding the incremental nature of the approach and not fully-convincing experimental results remain.

There are values in the work even though it appears to be a combination of several existing approaches. Given that open-vocabulary universal segmentation is a relatively new area and a lot of results reported in the original submission and rebuttal, the work has value.
Decision: Accept (poster)