SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition to the instance-level segmentation, we can also perform part-aware hierarchical segmentation. We concatenate the instance class names and part class names as labels. Some examples are "human ear", and "cat head". In the training process, we supervise the classification head with both part labels and instance labels. Specifically, we replace  $L_{cls}$  with  $L_{clsPart} + L_{clsThing}$  in Eq. (3). We combine parts segmentation and instance segmentation of the same image to get part-aware instance segmentation. Additionally, layers of hierarchy is obtained by grouping the parts. For example, the "head" consists of ears, hair, eyes, nose, etc. Fig. 5 illustrates this process. Fig. A1 highlights the difference of our approach with other methods.
3.8 Class-aware part segmentation with SAM
We can also perform the class-aware hierarchical segmentation by combining our semantic output with class-agnostic masks produced by SAM [24]. Specifically, given semantic masks  $M$ , their class probability  $P_{M}$ , and SAM-generated part masks  $S$ , we compute the class probability of mask  $S_{i} \in S$  with respect to class  $j$  as
$$
P _ {S} \left(S _ {i}, j\right) \propto \sum_ {M _ {k} \in M} P _ {M} \left(M _ {k}, j\right) \left| M _ {k} \cap S _ {i} \right| \tag {5}
$$
Where  $|M_k \cap S_i|$  is the area of intersection between mask  $M_k$  and  $S_i$ . We combine our semantic output with SAM because our pretraining datasets only contains object-centric masks, whereas the SA-1B dataset used by SAM contains many local segments and object parts.
Figure 6: Qualitative Analysis of Open Vocabulary Hierarchical Segmentation. Because of our hierarchical design, our model produces better-quality masks. In particular, our model can generalize to novel hierarchies that do not exist in part segmentation datasets.
Method	Backbone	COCO	ADE20K	PAS-P
PQ	APmask	APbox	mIoU	PQ	APmask	APbox	mIoU	mIoUPartS
MaskCLIP [10]	ViT16	-	-	-	-	15.1	6.0	-	23.7	-
X-Decoder [66]	FocalT	52.6	41.3	-	62.4	18.8	9.8	-	25.0	-
X-Decoder	DaViT-B	56.2	45.8	-	66.0	21.1	11.7	-	27.2	-
SEEM [67]	FocalT	50.6	39.5	-	61.2	-	-	-	-	-
SEEM	DaViT-B	56.2	46.8	-	65.3	-	-	-	-	-
ODISE [56]	ViT-H+SD	55.4	46.0	46.1	65.2	22.6	14.4	15.8	29.9	-
JPPF [21]	EffNet-b5	-	-	-	-	-	-	-	-	54.4
PPS [5]	RNST269	-	-	-	-	-	-	-	-	58.6
HIPIE	RN50	52.7	45.9	53.9	59.5	18.4	13.0	16.2	26.8	57.2
HIPIE	ViT-H	58.0	51.9	61.3	66.8	20.6	15.0	18.7	29.0	63.8
Table 2: Open-vocabulary panoptic segmentation (PQ), instance segmentation  $(\mathrm{AP}^{\mathrm{mask}})$ , semantic segmentation (mIoU), part segmentation  $(\mathrm{mIoU}_{\mathrm{PartS}})$ , and object detection  $(\mathrm{AP}^{\mathrm{box}})$ . N/A: not applicable. -: not reported.
We comprehensively evaluate HIPIE through quantitative and qualitative analyses to demonstrate its effectiveness in performing various types of open-vocabulary segmentation and detection tasks. The implementation details of HIPIE are explained in Sec. 4.1. Sec. 4.2 presents the evaluation results of HIPIE. Additionally, we conduct an ablation study of various design choices in Sec. 4.3.
Model Learning Settings can be found in our appendix materials.
Evaluation Metrics. Semantic Segmentation performance is evaluated using the mean Intersection-Over-Union (mIoU) metric. For Part segmentation, we report mIoUPartS, which is the mean IoU for part segmentation on grouped part classes [5]. Object Detection and Instance Segmentation results are measured using the COCO-style evaluation metric - mean average precision (AP) [34]. Panoptic Segmentation is evaluated using the Panoptic Quality (PQ) metric [23]. Referring Image Segmentation (RIS) [19, 60] is evaluated with overall IoU (oIoU).
Panoptic Segmentation. We examine Panoptic Quality (PQ) performance across MSCOCO [34] for closed-set and ADE20K [64] for open-set zero shot transfer learning. Based on Table 3 our model is able to outperform the previous close-set state-of-the-art using a ViT-H backbone by +1.8. In addition, we match the best open-set PQ results, while being able to run on more tasks and having a simpler backbone than ODISE [56]. Semantic Segmentation. The evaluation of our model's performance on various open-vocabulary semantic segmentation datasets is presented in Table 4. These datasets include: 1) A-150: This dataset comprises 150 common classes from ADE20K [64]. 2) A-847: This dataset includes all 847 classes from ADE20K [64]. 3) PC-59: It consists of 59 common classes from Pascal Context [39]. 4) PC-459: This dataset encompasses the full 459 classes of Pascal Context [39]. 5) PAS-21: The vanilla Pascal VOC dataset [12], containing 20 foreground classes and 1 background class. These diverse datasets enable a comprehensive evaluation of our model's performance across different settings, such as varying class sizes and dataset complexities. Table 4 provides insights into how our model performs in handling open-vocabulary semantic segmentation tasks, demonstrating
A-150
PQ	APmask	APbox	mIoU	mIoU	mIoU	APmask
19.7	15.0	17.7	23.4	-	-	36.1
21.8	13.1	-	29.6	9.2	16.1	32.2
8.9	14.9	11.9	6.4	1.8	5.8	42.1
18.1	16.7	20.2	19.8	4.8	12.2	41.0
22.9	19.0	22.9	29.0	9.7	14.4	41.6
Method
Data
OpenSeed	O365,COCO
X-Decoder	COCO,CC3M,SBU-C,VG,COCO-Caption,(Florence)
UNINEXT	O365,COCO,RefCOCO
HIPIE w/o CLIP	O365,COCO,RefCOCO,PACO
HIPIE w/ CLIP	+ (CLIP)
Table 3: Open-Vocabulary Universal Segmentation. We compare against other universal multi-task segmentation models. (*) denotes pretraining dataset of representations.
Method	A-150	PC-59	PAS-21	COCO
ZS3Net [2]	-	19.4	38.3	-
LSeg+ [26, 16]	18.0	46.5	-	55.1
HIPIE	26.8	53.6	75.7	59.5
vs. prev. SOTA	+7.1	+10.7	+28.3	+4.4
GroupViT [54]	10.6	25.9	50.7	21.1
OpenSeg [16]	21.1	42.1	-	36.1
MaskCLIP [10]	23.7	45.9	-	-
ODISE [56]	29.9	57.3	84.6	65.2
HIPIE	29.0	59.3	83.3	66.8
vs. prev. SOTA	-0.9	+2.0	-1.3	+1.6
Table 4: Comparison on open-vocabulary semantic segmentation. Baseline results are copied from [56].
Decoder	Fusion (things)	Fusion (stuff)	PQ APmask	oIOU
Unified	45.1 42.9	67.1
Decoupled	50.6 43.6	67.6
Unified (Fig. 4a)	✓	✓	44.6 42.5	66.8
Decoupled (Fig. 4b)	✓	✓	50.0 44.4	77.1
Decoupled (Fig. 4c)	✓	51.3 44.4	77.3
Table 5: An ablation study on different decoder and text-image fusion designs, as depicted in Fig. 4. We report PQ for panoptic segmentation on MSCOCO,  $\mathrm{AP^{mask}}$  for instance segmentation on MSCOCO, and oIoU for referring segmentation on RefCOCO's validation set. Our final choice is highlighted in gray.
its effectiveness and versatility in detecting and segmenting a wide range of object categories in real-world scenarios.
Part Segmentation. We evaluate our models performance on Pascal-Panoptic-Parts dataset [5] and report  $\mathrm{mIoU}_{\mathrm{partS}}$  in Table 3. We followed the standard grouping from [5]. Our model outperforms state-of-the-art by  $+5.2$  in this metric. We also provide qualitative comparisons with Grounding DINO + SAM in Fig. 7. Our findings reveal that the results of Grounding SAM are heavily constrained by the detection performance of Grounding DINO. As a result, they are unable to fully leverage the benefits of SAM in producing accurate and fine-grained part segmentation masks.
Raw Image
SAM Mask
GDINO+SAM
Figure 7: Results of merging HIPIE with SAM for class-aware image segmentation on SA-1B dataset. Grounded-SAM (Grounding DINO + SAM) [29, 24] cannot fully leverage the benefits of SAM in producing accurate and fine-grained part segmentation masks. Our method demonstrates fewer misclassifications and overlooked masks across the SA-1B dataset compared to the Grounded-SAM approach.
HIPIE+SAM
Method	Backbone	Object Detection
AP	APS	APM	APL
Deform. DETR [65]	RN50	46.9	29.6	50.1	61.6
DN-DETR [27]	RN50	48.6	31.0	52.0	63.7
UNINEXT [58]	RN50	51.3	32.6	55.7	66.5
HIPIE	RN50	53.9	37.5	58.0	68.0
vs. prev. SOTA	+2.6	+4.9	+2.3	+1.5
Cas. Mask-RCNN [3]	CNeXtL	54.8	-	-	-
ViTDet-H [31]	ViT-H	58.7	-	-	-
UNINEXT [58]	ViT-H	58.1	40.7	62.5	73.6
HIPIE	ViT-H	61.3	45.8	65.7	75.9
vs. prev. SOTA	+3.2	+5.1	+3.2	+2.3
Table 6: Comparisons on the instance segmentation and object detection tasks. We evaluate model performance on the validation set of MSCOCO.
Method	Backbone	COCO	COCO+	COCOg
oIoU	oIoU	oIoU
MAttNet [60]	RN101	56.5	46.7	47.6
VLT [9]	Dark56	65.7	55.5	53.0
RefTR [40]	RN101	74.3	66.8	64.7
UNINEXT [58]	RN50	77.9	66.2	70.0
UNINEXT [58]	ViT-H	82.2	72.5	74.7
HIPIE	RN50	78.3	66.2	69.8
HIPIE	ViT-H	82.6	73.0	75.3
vs. prev. SOTA	+0.4	+0.5	+0.6
Table 7: Comparison on the referring image segmentation (RIS) task. We evaluate the model performance on the validation sets of RefCOCO, RefCOCO+, and RefCOCOg datasets using overall IoU (oIoU) metrics.
Object Detection and Instance Segmentation. We evaluate our model's object detection and instance segmentation capabilities following previous works [28, 67, 56]. On MSCOCO [34] and ADE20K [64] datasets, HIPIE achieves an increase of  $+5.1$  and  $+0.6\mathrm{AP}^{\mathrm{mask}}$  respectively. Detailed comparisons are provided in Sec. 4.2 which demonstrate state-of-the-art results on ResNet and ViT architectures consistently across all Average Precision metrics.
Referring Segmentation. Referring image segmentation (RIS) tasks are examined using the Ref-COCO, RefCOCO+, and RefCOCOg datasets. Our model outperforms all the other alternatives by an average of +0.5 in overall IoU (oIoU).
To demonstrate the effectiveness of our design choices for text-image fusion mechanisms and representation learning modules for stuff and thing classes, we conduct an ablation study (depicted in Fig. 4) and present the results in Table 5. From this study, we draw several important conclusions: 1) Text-image fusion plays a critical role in achieving accurate referring segmentation results. 2) The early text-image fusion approach for stuff classes negatively impacts the model's performance on panoptic segmentation. This finding validates our analysis in the introduction section, where we highlighted the challenges introduced by the high levels of confusion in stuff's textual features, which can adversely affect the quality of representation learning. 3) Our design choices significantly improve the performance of panoptic segmentation, instance segmentation, and referring segmentation tasks. These conclusions underscore the importance of our proposed design choices in achieving improved results across multiple segmentation tasks.
This paper presents HIPIE, an open-vocabulary, universal, and hierarchical image segmentation model that is capable of performing various detection and segmentation tasks using a unified framework, including object detection, instance-, semantic-, panoptic-, hierarchical-(whole instance, part, subpart), and referring-segmentation tasks. Our key insight is that we should decouple the representation learning modules and text-image fusion mechanisms for background (i.e., referred to as stuff) and foreground (i.e., referred to as things) classes. Extensive experiments demonstrate that HIPIE achieves state-of-the-art performance on diverse datasets, spanning across a wide range of tasks and segmentation granularity.
Acknowledgement Trevor Darrell and XuDong Wang were funded by DoD including DARPA LwLL and the Berkeley AI Research (BAIR) Commons.
[61] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[62] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022.
[63] W. Zhao, Y. Rao, Z. Liu, B. Liu, J. Zhou, and J. Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153, 2023.
[64] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[65] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[66] X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270, 2022.
[67] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718, 2023.
A.1 List of Datasets
semantic	instance	panoptic	grounding	part	training	# images
ADE-150	✓	✓	✓	2000
Pascal VOC	✓	1449
Pascal Context-59	✓	5105
Pascal-Panoptic-Parts	✓	✓	✓	✓	*	10103
COCO	✓	✓	✓	✓	121408
RefCOCO	✓	✓	19994
RefCOCO+	✓	✓	19992
RefCOCOg	✓	✓	26711
Table A1: List of the dataset used. The checkmarks denote whether a dataset has a particular type of annotation and whether the dataset is used in the training process. * Because of a data leak between Pascal-Panoptic-Parts and other Pascal datasets, we use weights trained without Pascal-Panoptic-Parts in those evaluations unless otherwise specified.
We report the statistics of datasets used in training and evaluation in table Table A1. Additionally, we further evaluate our model on 35 object detection datasets and 25 segmentation datasets in Sec. A.4.2. In total, we benchmarked our model on around 70 datasets. These benchmarks show our model can adapt to many different scenarios and retain a reasonable performance in a zero-shot manner.
A.2 Hierarchical Segmentation
Figure A1: Hierarchical Design of HIPIE compared with other methods.
Fig. A1 highlights the difference of our approach with other methods for hierarchical segmentation. We concatenate class names of different hierarchies as prompts. During the training, we uniquely contrast a mask embedding with both scene-level and part-level labels explicitly. Previous works such as UNINEXT and ODISE only treat these classes as normal multi-word labels. While UNINEXT allows contrasting different words individually because of the design of BERT encoder, it leads to suboptimal signals. In the example above, "person head" has both positive and negative target for "person".
A.3 Experiment Setup
A.3.1 Model Learning Settings
HIPIE is first pre-trained on Objects365 [48] for 340k iterations, using a batch size of 64 and a learning rate of 0.0002, and the learning rate is dropped by a factor of 10 after the 90th percentile of the schedule. After the pre-training stage, we finetune HIPIE on COCO [34], RefCOCO, RefCOCOg, and RefCOCO+ [41, 61] jointly for 120k iterations, using a batch size of 32 and a learning rate of 0.0002. For both stages, we resize the original images so that the shortest side is at least 800
pixels and at most 1024 pixels, while the longest side is at most 1333 pixels. For part segmentation, we train additionally train our model jointly on Pascal-Panoptic-Parts [5] dataset and all previously mentioned datasets. Because of potential data leaks between Pascal-Panoptic-Parts and other Pascal datasets used in the open-vocabulary segmentation evaluation, we report those numbers with weights not trained on Pascal-Panoptic-Part dataset. Because of our hierarchical design, our model produces better-quality masks. In particular, our model can generalize to novel hierarchies that do not exist in part segmentation datasets. In Fig. 6, we provide visualization of such results.
A.3.2 Implementation Details
For loss functions in Eq. (3), we have  $\lambda_{cls} = 2.0$ ,  $\lambda_{mask} = 5.0$ ,  $\lambda_{box} = 5.0$ ,  $\lambda_{ce} = 1.0$ ,  $\lambda_{dice} = 1.0$ ,  $\lambda_{L1} = 1.0$ ,  $\lambda_{giou} = 0.2$ . For  $\lambda$  in Eq. (4), we use  $\lambda = 0.2$  for seen classes during the training and  $\lambda = 0.45$  for novel classes. In close-set evaluation, we set  $\lambda = 0.0$  and do not use CLIP. We also do not use CLIP for PAS-21 evaluation (whose classes are mostly covered by COCO) because we find it degrades the performance. We use 800 and 1024-resolution images during the training. For evaluations, we use 1024-resolution images.
A.3.3 Training Process
Stage	Task	Dataset	Batch Size	Max Iter	Step
I	OD&IS	Objects365	64	340741	312346
II	OD&IS	COCO	32	91990	76658
REC&RIS	RefCOCO/g/+	32
III	PanoS	COCO	32
REC&RIS	RefCOCO/g/+	32	150000	100000,135000
PartS	Pascal-Panoptic-Parts	32
Table A2: Training Process. Following UNINEXT [58], We first pretrain our model for object detection on Object365 for 340k iteration (Stage I). Then we fine-tune our model jointly on COCO for object detection, instance segmentation, referring expression comprehension (REC), and referring segmentation (RIS) for 92k iteration (Stage II). We further jointly train our model on Panoptic Segmentation, REC, RIS, and Part Segmentation for 150k iteration (Stage III)
We train all our models on NVIDIA-A100 GPUs with a batch size of 2 per GPU using AdamW [38] optimizer. We use a base learning rate of 0.0001 and a weight decay of 0.01. The learning rate of the backbone is further multiplied by 0.1. Following UNINEXT [58], We first pretrain our model for object detection on Object365 for 340k iteration (Stage I). Then we fine-tune our model jointly on COCO for object detection, instance segmentation, referring expression comprehension (REC), and referring segmentation (RIS) for 91k iteration (Stage II). We further jointly train our model on Panoptic Segmentation, REC, RIS, and Part Segmentation for 150k iteration (Stage III). In Stage I, the learning rate is dropped by a factor of 10 after 312k iterations. In stage II, the learning rate is dropped by a factor of 10 after 77k iterations. In Stage III, the learning rate is dropped by a factor of 10 after 100k and 135k iterations. In all stages, we sample uniformly across datasets when there are multiple datasets. The global batch size is 64 in Stage I and 32 in Stage II and III. Notably, our stage I and II is identical to the setup of UNINEXT. For ablation studies, we train stage III only and reduce the schedule to 90k iterations. The learning rate schedule is also scaled accordingly. The details of training recipe is shown in Table A2.
A.4 Additional Evaluations
A.4.1 Referring Expression Comprehension
In addition to Referring Segmentation reported in Table 7, we further report results on Referring Expression Comprehension (REC), which aims to locate a target object in an image at the pixel-level, given a referring expression as input. We establish new state-of-the-art performance by an average of  $+0.3\mathrm{P@0.5}$  and  $+0.5$  oIoU across three datasets.
Method	Backbone	RefCOCO	RefCOCO+	RefCOCOg
oIoU	P@0.5	oIoU	P@0.5	oIoU	P@0.5
MAttNet [60]	RN101	56.5	76.7	46.7	65.3	47.6	66.6
VLT [9]	Dark56	65.7	76.2	55.5	64.2	53.0	61.0
RefTR [40]	RN101	74.3	85.7	66.8	77.6	64.7	82.7
UNINEXT [58]	RN50	77.9	89.7	66.2	79.7	70.0	84.0
UNINEXT [58]	ViT-H	82.2	92.6	72.5	85.2	74.7	88.7
HIPIE	RN50	78.3	90.1	66.2	80.0	69.8	83.6
HIPIE	ViT-H	82.6	93.0	73.0	85.5	75.3	88.9
vs. prev. SOTA	+0.4	+0.4	+0.5	+0.3	+0.6	+0.2
Table A3: Comparison on the referring expression comprehension (REC), and referring image segmentation (RIS) tasks. The evaluation is carried out on the validation sets of RefCOCO, RefCOCO+, and RefCOCOg datasets using Precision@0.5 and overall IoU (oIoU) metrics for REC and RIS, respectively.
Method	Mean	Median	AerialMaritime	DraianLarge	AerialMaritime	DraianTied	American Sign Lang Letters	Aquarium	BCCB	bogboeboards	brackish	Underwater	ChessPieces	Cottonwall	Rabbits	dice	mediumColor	EggControls	EgoHands specific	HardHat	Workers	Mask Wearing	Mountain Dew Commercial North Mushrooms open Poetry Vision Oxford Pets by- breed Oxford Pets by-species	Packs	Pascal VOC	Pistols	PKLot	planloc	Pothole	Raccoon selfdriving Car OpenImages Thermal Cheetah thermal Dogs And People Un Cards Vehicles OpenImages website	Spectacles	Wildfire Smoke
MDETR	10.7	3.0	0.6	5.4	0.3	1.7	6.7	0.0	0.7	3.0	0.6	5.0	0.0	3.8	5.9	3.5	0.4	0.4	3.0	3.9	8.8	0.0	0.0	0.7	63.6	5.6	15.9	0.0	512.750.62.8	8.1	4.5	42.80	0.013.4	0.7	12.5
GLIP-T	11.4	1.6	8.3	17.1	0.1	1.6	0.0	1.7	0.0	0.7	0.0	0.5	0.1	1.1	0.1	0.1	2.7	0.6	15.3	5.9	0.0	0.3	1.6	58.3	35.1	23.1	60.0	0.1	6.6	6.2	7.4	15.9	0.2	3.8	78.0	0.055.0	0.3	0.0
Table A4: We present the object detection results in the OdinW benchmark. We report mAP and mean results averaged over 35 datasets. Notably, our ResNet-50 baseline surpasses GLIP-T by +3.1. We use the notation HIPIE† and HIPIE‡ to denote our method with ResNet-50 and ViT-H backbones, respectively.
Method	Mean	Median	Airplane Parts	Bottles	Brain Tumor	Chicken	Cows	Electric Shaver	Elephants	Fruits	Garbage	Ginger Garlic	Hand	Hand Metal House Parts	HouseHold items	Nutterfly Squireel	Phones	Poles	Puppies	Rail Salmon Fillet	Syr烟草 Tablets	Toilets	Trash	Watermelon
X-Decoder(L)	32.3	22.3	13.1	42.1	2.2	8.6	44.9	7.5	66.0	79.2	33.0	11.6	75.9	42.1	7.0	53.0	68.4	15.6	20.1	59.0	2.3	19.0	67.1	22.5	9.9	22.3	13.8
HIPIE (H)	41.2	45.1	14.0	45.1	1.9	46.5	50.1	76.1	68.6	61.1	31.2	24.3	94.2	64.0	6.8	53.4	79.7	7.0	6.7	64.6	2.2	41.8	81.5	8.8	17.9	31.2	50.6
Table A5: Segmentation Result on SeginW benchmark across 25 datasets. We report mAP. We outperform X-Decoder by a large margin (+8.9)
A.4.2 Object Detection and Segmentation in the Wild
To further examine the open-vocabulary capability of our model, we evaluate it on the Segmentation in the Wild (SeginW) [66] consisting of 25 diverse segmentation datasets and Object Detection in the Wild (OdinW) [29] Benchmark consisting of 35 diverse detection datasets. Since OdinW benchmark contains Pascal VOC and some of the classes in SeginW benchmark are covered by Pascal-Panoptic-Parts, we use a version of our model that is not trained on Pascal-Panoptic-Parts for both benchmarks for a fair comparison.
We report the results in Table A5 and Table A4. Notably, our method establishes a new state-of-the-art of SeginW benchmark by an average of  $+8.9\mathrm{mAP}$  across 25 datasets. We achieve comparable performance under similar settings. In particular, our ResNet-50 baseline outperforms GLIP-T by  $+3.1\mathrm{mAP}$ . We note that other methods such as GroundingDINO [36] achieve better absolute performance by introducing more grounding data, which can be critical in datasets whose classes are not common objects. (For example, the classes of Boggle Boards are letters, the classes of UnoCards are numbers, and the classes of websiteScreenshot are UI elements).
A.5 Other Ablation Studies
We provide further ablations on a few design choices in this section.
Text Encoder. We experiment with replacing the BERT text encoder in UNINEXT with a pre-trained CLIP encoder. Additionally, following practices of ODISE [56], we prompt each label to a sentence "a photo of <label>" For RIS and REC tasks, the language expression remains unchanged. We report
COCO	RefCOCO
PQ	APMask	oIoU
CLIP	51.5	44.3	48.7
BERT	51.3	44.4	77.3
Table A6: Ablation Studies on the choice of Text Encoder. We find that while CLIP and BERT achieve similar performance on panoptic and instance segmentation, BERT performs significantly better on Referring Instance Segmentation (+28.6 oIoU).
COCO	RefCOCO
PQ	APMask	oIoU
w/o OTA	50.9	43.6	76.3
w/ OTA	51.3	44.4	77.3
Table A7: Ablation Studies on the SimOTA matching process. Introducing SimOTA leads to performance improvement in all evaluation metrics.
the result in Table A6. We find that while CLIP and BERT achieve similar performance on panoptic and instance segmentation, BERT performs significantly better on referring instance segmentation  $(+28.6\mathrm{oIoU})$ . We hypothesize that this may be caused by the lack of explicit language-focused training which can help achieve a better understanding of referring expression.
SimOTA.Following UNINEXT [58] we adopted simOTA in the matching process for "thing" classes during the training. We experiment with removing simOTA matching and use standard one-to-one matching instead. We report the result in Table A7. We find that simOTA improves the performance on both panoptic segmentation and referring instance segmentation.
A.6 Limitations