SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation Setup for Three Task Settings. Since the state-of-the-art object-level OVSS models are powerful, we design two evaluation settings to investigate the primary bottlenecks: (1) Oracle-Obj Setting: The ground-truth mask and class of objects are assumed to be known. (2) Pred-Obj Setting: The ground-truth mask and class of objects are not provided.
Implementation Details. The details of baselines and training are left to the supplementary material.
5.2 Results in Generalized Zero-Shot Part Segmentation
Settings. We reported results of both two-stage and one-stage baselines on Pascal-Part-116 and ADE20K-Part-234 datasets as shown in Table 1 and Table 2 respectively. Firstly, to provide a comprehensive comparison, we reported the results of the supervised baseline: MaskFormer [5] with ResNet-50 backbone on the seen and unseen classes. Then for the two-stage baselines, we reported the results of the original ZSseg [33] and our modified ZSseg (ZSseg+) with different finetuning methods for the second stage: Compositional Prompt Tuning based on CoOp [38] and CoCoOp [37], abbreviated to CPT-CoOp and CPT-CoCoOp. For the one-stage baselines, we examined two state-of-the-art methods: CAT-Seg [7] and CLIP-Seg [22]. We used the pretrained model of CATSeg trained on COCO-Stuff and CLIPSeg trained on PhraseCut. Then we reported their results without any finetuning, as well as the results when adopting various finetuning strategies. For the Pred-Obj Setting evaluation, we use ZSseg's pretrained object model on COCO-Stuff in the two-stage baselines while don't adopt separate object models in one-stage baselines.
Two-Stage Baselines. We evaluated the effects of the Object Mask Prompt, Mask Denoise, and CPTCoOp in ZSseg+ on Pascal-Part-116 as shown in Table 3. We gradually add each approach to ZSseg to measure its individual effect under the Oracle-Obj Setting. The results indicate that the
Table 1: Zero-shot performance of the two-stage and one-stage baselines on Pascal-Part-116.
Model	Backbone	Finetuning	Oracle-Obj	Pred-Obj
Seen	Unseen Harmonic	Seen	Unseen Harmonic
Fully-Supervised Baseline
MaskFormer [5]	ResNet-50	X	55.28	52.14	-	53.07	47.82	-
Two-Stage Baselines
ZSseg [33]	ResNet-50	X	49.35	12.57	20.04	40.80	12.07	18.63
ZSseg+	ResNet-50	CPTCoOp	55.33	19.17	28.48	54.23	17.10	26.00
ZSseg+	ResNet-50	CPTCoCoOp	54.43	19.04	28.21	53.31	16.08	24.71
ZSseg+	ResNet-101c	CPTCoOp	57.88	21.93	31.81	56.87	20.29	29.91
One-Stage Baselines
CATSeg [7]	ResNet-101 & ViT-B/16	X	14.89	10.29	12.17	13.65	7.73	9.87
CATSeg [7]	ResNet-101 & ViT-B/16	B+D	43.97	26.11	32.76	41.65	26.08	32.07
CLIPSeg [22]	ViT-B/16	X	22.33	19.73	20.95	14.32	10.52	12.13
CLIPSeg [22]	ViT-B/16	VA+L+F+D	48.68	27.37	35.04	44.57	27.79	34.24
Table 2: Zero-shot performance of the two-stage and one-stage baselines on ADE20K-Part-234.
Model	Backbone	Finetuning	Oracle-Obj	Pred-Obj
Seen	Unseen Harmonic	Seen	Unseen Harmonic
Fully-Supervised Baseline
MaskFormer [5]	ResNet-50	X	46.25	47.86	-	35.52	16.56	-
Two-Stage Baselines
ZSseg+	ResNet-50	CPTCoOp	43.19	27.84	33.85	21.30	5.60	8.87
ZSseg+	ResNet-50	CPTCoCoOp	39.67	25.15	30.78	19.52	2.98	5.17
ZSseg+	ResNet-101c	CPTCoOp	43.41	25.70	32.28	21.42	3.33	5.76
One-Stage Baselines
CATSeg [7]	ResNet-101 & ViT-B/16	X	11.49	8.56	9.81	6.30	3.79	4.73
CATSeg [7]	ResNet-101 & ViT-B/16	B+D	31.40	25.77	28.31	20.23	8.27	11.74
CLIPSeg [22]	ViT-B/16	X	15.27	18.01	16.53	5.00	3.36	4.02
CLIPSeg [22]	ViT-B/16	VA+L+F+D	38.96	29.65	33.67	24.8	6.24	9.98
Table 3: Ablations on proposed strategies in ZSseg+ by adding each module to ZSseg on Pascal-Part-116.
Model	Oracle-Obj
Seen	Unseen	Harmonic
ZSseg [33]	49.35	12.57	20.04
+Obj Mask Prompt	48.00	13.89	21.54
+Mask Denoise	48.00	16.84	24.93
+CPTCoOp	55.33	19.17	28.48
Table 4: Cross-Dataset Performance. Models are trained on the source dataset ADE20K-Part-234 and tested on the target dataset Pascal-Part-116.
Model	Source	Target
Oracle Pred	Oracle Pred
CATSeg	27.95	17.22	16.00	14.72
VA+L+F	35.01	21.74	16.18	11.70
VA+L+F+D	37.76	21.87	19.69	13.88
Object Mask Prompt improves the performance on unseen classes  $(+1.32\%)$ , and adding Mask Denoise further improves it  $(+2.95\%)$ . However, there is a slight performance drop  $(-1.35\%)$  on the seen classes. Moreover, finetuning the CLIP embeddings with CPTCoOp leads to significant performance gains on both the seen  $(+7.33\%)$  and unseen  $(+2.33\%)$  classes. Notably, the finetuning with CPTCoOp requires only 500 iterations and less than 128 samples per class.
One-Stage baselines. Both CATSeg and CLIPSeg employ a frozen CLIP visual encoder to extract image features. But CATSeg further uses an extra learnable visual backbone to guide the pixel decoder. In Table 1 and Table 2, we can see that the CATSeg and CLIPSeg models, pretrained on object datasets, already show impressive results. Notably, CLIPSeg even outperforms ZSseg+ with a ResNet-50 backbone on the unseen classes. The comparison between CLIPSeg and CATSeg shows that OVSS models trained with phrases of objects exhibit stronger generalization ability to parts than sole objects. Hence, we explore alternative finetuning strategies rather than training from scratch. We finetune three different components: language embedding layer in text encoder (L), FiLM (F) and Decoder (D) of CLIPSeg and two added lightweight modules: CLIP-Adapter [12] to the visual encoder (VA) and text encoder (TA). As shown in Table 5, we draw three key conclusions: (1) Multi-modality finetuning is better than single-modality (VA+L+FiLM surpasses VA+FiLM and L+FiLM); (2) Finetuning language embedding performs better than the text adapter (VA+L+F surpasses VA+TA+F); (3) Parameter-efficient finetuning can effectively transfer the knowledge from large-scale foundation models and object-level datasets to part parsing. We can see that VA+L+F even outperforms finetuning the entire pixel decoder on the unseen classes.
The qualitative results, which present a comparison among ZSseg+, CATSeg and CLIPSeg on the unseen class "Bird" of Pascal-Part-116, as well as additional qualitative results on Pascal-Part-116 and ADE20K-Part-234, are available in Section C of the supplementary material.
5.3 Results in Few-Shot Part Segmentation
In the few-shot setting, the two-stage baselines have inherent limitations. Firstly, training the MaskFormer from scratch is prone to overfitting with limited data. Secondly, the class-agnostic proposal generation is unable to generalize from object to part, making it hard to make use of the
Table 5: Performance on finetuning various modules of CLIPSeg in both the zero-shot setting and few-shot setting on Pascal-Part-116 and ADE20K-Part-234.
Model	Pascal-Part-116	ADE20K-Part-234
Setting	Zero-Shot Setting (Oracle)	Few-Shot Setting	Few-Shot Setting
Finetuning	Seen	Unseen	Harmonic	Oracle	Pred	Oracle	Pred
X	22.33	19.73	20.95	21.58	-	15.38	-
D	44.65	26.03	32.89	29.86	27.16	24.01	12.96
F	31.34	21.45	25.46	-	-	-
L+F	38.11	21.14	27.19	27.61	25.90	26.04	14.55
TA+F	33.13	21.70	26.23	-	-	-
VA+F	45.13	22.55	30.07	29.99	26.84	23.74	12.72
VA+TA+F	44.83	24.16	31.39	-	-	-
VA+L+F	47.04	27.69	34.85	31.34	27.67	28.67	17.32
VA+L+F+D	48.68	27.37	35.04	33.13	30.70	29.36	16.12
object-level data. We reported the results of finetuning various modules in CLIPSeg as shown in Table 5. We can draw similar conclusions regarding the different finetuning strategies as observed in the zero-shot setting results. Besides, we observe inconsistent performance trends on Pascal-Part-116 and ADE20K-Part-234: (1) Visual modality finetuning is superior on Pascal-Part-116 while it is inferior on ADE20K-Part-234  $(\mathbf{L} + \mathbf{F}$  v.s.  $\mathbf{V}\mathbf{A} + \mathbf{F})$ . (2) Finetuning decoder  $(\mathbf{D})$  even underperforms the single language modality  $(\mathbf{L} + \mathbf{F})$  on ADE20K-Part-234, whereas the opposite trend is observed on Pascal-Part-116. (3) Finetuning  $\mathbf{V}\mathbf{A} + \mathbf{L} + \mathbf{F} + \mathbf{D}$  on ADE20K-Part-234 brings  $0.69\%$  gains to  $\mathbf{V}\mathbf{A} + \mathbf{L} + \mathbf{F}$  under the Oracle-Obj Setting but causes  $1.2\%$  performance drops  $(1.2\%)$  under the Pred-Obj Setting, which is different from the Pascal-Part-116. The cause of the inconsistencies is that, ADE20K-Part-234 is more challenging than Pascal-Part-116 thus the transferability gap from object to part is larger. Finetuning more parameters may cause worse performance and is biased to parts compared to objects. Qualitative results are left to Section C of the supplementary material.
5.4 Results in Cross-Dataset Part Segmentation
We reported three baselines as shown in Table 4: CATSeg with backbone and decoder finetuned, CLIPSeg with Visual Adapter, Language Embedding and FiLM (VA+L+F) finetuned and further the decoder  $(\mathbf{VA} + \mathbf{L} + \mathbf{F} + \mathbf{D})$  finetuned. Under the Oracle-Obj Setting, CLIPSeg performs better than CATSeg on both source and target datasets. But CATSeg is less biased to the part as it performs the best on the target dataset under the Pred-Obj Setting. Hence CATSeg is less prone to overfitting than CLIPSeg with even more learnable parameters. We also explore the cross-dataset part segmentation from Pascal-Part-116 to ADE20K-Part-234 for analyzing the potential failures cases in either the part boundaries delineation or the language misunderstanding. The qualitative results are left to Section C of the supplementary material.
In conclusion, open-vocabulary part segmentation has unique challenges: the limited availability of labeled data, the complexity of part structures and the open granularity challenge. To inspire future research, we proposed the OV-PARTS benchmark with newly cleaned Pascal-Part-116 and ADE20K-Part-234 datasets. And we introduced three benchmark tasks: Generalized Zero-Shot Segmentation, Cross-Dataset Segmentation, and Few-Shot Segmentation, which assesses the analogical reasoning, open granularity, and few-shot adapting abilities of OV-PARTS models. Furthermore, we improved two paradigms from existing object-level OVSS methods as the baselines. Through comprehensive experimental analysis, we provided insights into the strengths and limitations of current approaches, highlighting the potential of leveraging large foundation models for OV-PARTS.
Acknowledgements. This work is supported by Shanghai Artificial Intelligence Laboratory, HKU Startup Fund, HKU Seed Fund for Basic Research, and HKU Seed Fund for Translational and Applied Research. Shu Kong is supported by SRG2023-00044-FST.
[38] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[39] Tianfei Zhou, Wenguan Wang, Si Liu, Yi Yang, and Luc Van Gool. Differentiable multi-granularity human representation learning for instance-aware human semantic parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1622-1631, 2021.
Table A.1: Model complexity analysis on Pascal-Part-116. ZSseg+ 1/2st is first/second stage.
Model	Backbone	Image Size	Trainable Params (M)	FLOPs (G)
ZSseg+ 1st	ResNet-101c	512 × 704	61.1	103.9
ZSseg+ 2st	ViT-B/16	384 × 384	0.004	58.9
CATSeg	ResNet-101 & ViT-B/16	384 × 384	30.9	139.0
CLIPSeg	ViT-B/16	352 × 352	1.5	97.9
Two-Stage Baselines. Except for the Object Mask Prompt and Compositional Prompt Tuning designs, we follow the default architecture in the original ZSseg paper. The number of part queries is set to 50. All the two-stage baselines are trained with AdamW optimizer with the initial learning rate of 1e-4 and weight decay of 1e-4. A poly learning rate policy with a power of 0.9 is adopted. The total batch size is 16 and the total training iteration is 20k. For the training of Compositional Prompt Tuning, a SGD optimizer with the initial learning rate of 2e-2 and weight decay of 5e-4 is used. And we adopt a warm-up cosine learning rate policy with 100 warm-up iterations. The total batch size is 32 and the total training iteration is 3k. We sample 128 training samples for each object part class. The length of the learnable object and part prompt tokens are 4. The object tokens are initialized from the text embedding of the template "a photo of". The initial value of the learnable fusion weight is 0.5.
One-Stage Baselines. We adopt the original architecture of both CATSeg and CLIPSeg as described in their respective papers. For finetuning CATSeg, we utilize their pretrained model with a ResNet-101 backbone. However, while CATSeg achieves the best performance by finetuning the attention layers of CLIP's visual encoder in open vocabulary object segmentation, we observe poor performance with the same finetuning strategy in OV-PARTS. In our experiments, we only finetune the backbone with a backbone multiplier of 0.1 and the swim transformer based decoder. We employ an AdamW optimizer with an initial learning rate of 2e-4, weight decay of 1e-4, and a cosine learning rate policy. The total batch size is 8, and the training iterations amount to 40k. For CLIPSeg, we utilize the same optimizer settings and learning rate policy as CATSeg. The training iterations are set to 20k for the zero-shot/cross-dataset part segmentation setting and 3k for the few-shot part segmentation setting.
Model Complexity. We analyze the computational complexity of these two types of baselines and summarize the number of trainable parameters and FLOPs in Table A.1. The complexity is evaluated on Pascal-Part-116. It's evident that the one-stage CLIPSeg, which solely refines a lightweight transformer decoder and employs parameter-efficient modules, showcases the fewest trainable parameters and the lowest FLOPs. In contrast, the two-stage ZSseg+ approach, involving the training of a complete maskformer with a larger resolution, leads to the highest count of trainable parameters and FLOPs.
B Benchmark Datasets Details
B.1 Pascal-Part-116
Pascal-Part-116 contains 8431 training images and 850 testing images. Compared to the original version of Pascal-Part, we discard the directional indicator for some part classes and merge them to avoid overly intricate part definitions. The category vocabulary and merging details are as follows:
aeroplane [body, stern, left/right wing, tail, engine, wheel]
bicycle [front/back wheel, saddle, handlebar, chainwheel, headlight]
bird [left/right wing, tail, head, left/right eye, beak, torso, neck, left/right leg, left/right foot]
bottle [body, cap]
bus [wheel, headlight, front, left/right side, back, roof, left/right mirror, front/back license plate, door, window]
Figure A.1: The number of object masks that have corresponding part masks in Pascal-Part-116.
Figure B.2: The number of part masks for each object class in Pascal-Part-116. Each horizontal bar is color-coded to represent a specific part class belonging to the object. The colors of the bars are ordered from left to right based on the part sequence in the list of objects with parts.
car [wheel, headlight, front, left/right side, back, roof, left/right mirror, front/back license plate, door, window]
cat [tail, head, left/right eye, torso, neck, left-front/right-front/last-back/right-back leg, nose, left-front/right-front/left-back/right-back paw, left/right ear]
cow [tail, head, left/right eye, torso, neck, left-front-upper/left-front-lower /right-front-upper/right-front-lower/left-back-upper/left-back-lower/right-back-upper /right-back-lower leg, left/right ear, muzzle, left/right horn]
dog [tail, head, left/right eye, torso, neck, left-front/right-front/left-back/right-back leg, nose, left-front/right-front/left-back/right-back paw, left/right ear, muzzle]
horse [tail, head, left/right eye, torso, neck, left-front-upper/left-front-lower /right-front-upper/right-front-lower/left-back-upper/left-back-lower/right-back-upper /right-back-lower leg, left/right ear, muzzle, left-front/right-front/left-back /right-back hoof]
motorbike [front/back wheel, saddle, handlebar, headlight]
person [head, left/right eye, torso, neck, left-lower/right-lower/left-upper/right-upper leg, foot, nose, left/right ear, left/right eyebrow, mouth, hair, left/right lower arm, left/right upper arm, left/right hand]
pottedplant [pot, plant]
sheep [tail, head, left/right eye, torso, neck, left-front-upper/left-front-lower /right-front-upper/right-front-lower/left-back-upper/left-back-lower/right-back-upper /right-back-lower leg, left/right ear, muzzle, left/right horn]
train [headlight, head, front, left/right side, back, roof, coach]
tvmonitor [screen]
The unseen objects are colored blue and the removed terms are colored purple.
B.2 ADE20K-Part-234
The original subset of SceneParse150 comprises 20,210 training images and 2,000 validation images. After filtering out less frequent parts, the subset is reduced to 7,347 training images and 1,016 validation images. In ADE20K, most object parts have sparse mask annotations, and only a subset of object instances have part annotations. Hence, ADE20K-Part-234 provides the instance-level object mask annotations along with their part mask annotations. To maximize the use of labeled data and ensure authentic evaluations, different data splits are designed for the three task settings. (1) Generalized Zero-Shot Part Segmentation: Models are trained on the seen object instances from the 7,347 training images. Testing is performed on both unseen object instances from the same 7,347 training images and all object instances from the 1,016 validation images. (2) Few-Shot Part Segmentation: For each object class, 16 training images are sampled following the approach in Pascal-Part-116. we adapt the validation set from the generalized zero-shot part segmentation setting by removing the images that occur in the sampled 16-shot training set. (3) Cross-Dataset Part Segmentation: The original data split (7347/1016 training/validation images) is used since we mainly test on the Pascal-Part-116 dataset. The annotated objects with their parts are listed as follows:
person [arm, back, foot, gaze, hand, head, leg, neck, torso]  
door [door frame, handle, knob, panel]  
clock [face, frame]  
toilet [bowl, cistern, lid]  
cabinet [door, drawer, front, shelf, side, skirt, top]  
sink [bowl, faucet, pedestal, tap, top]  
lamp [arm, base, canopy, column, cord, highlight, light source, shade, tube]  
sconce [arm, backplate, highlight, light source, shade]  
chair [apron, arm, back, base, leg, seat, seat cushion, skirt, stretcher]  
chest of drawers [apron, door, drawer, front, leg]
Figure B.3: The statistics for the number of object masks with part masks on ADE20K-Part-234.
chandelier [arm, bulb, canopy, chain, cord, highlight, light source, shade] bed [footboard, headboard, leg, side rail]  
table [apron, drawer, leg, shelf, top, wheel]  
armchair [apron, arm, back, back pillow, leg, seat, seat base, seat cushion] ottoman [back, leg, seat]  
shelf [door, drawer, front, shelf]  
swivel chair [back, base, seat, wheel]  
fan [blade, canopy, tube]  
coffee table [leg, top]  
stool [leg, seat]  
sofa [arm, back, back pillow, leg, seat base, seat cushion, skirt]  
computer [computer case, keyboard, monitor, mouse]  
desk [apron, door, drawer, leg, shelf, top]  
wardrobe [door, drawer, front, leg, mirror, top]  
car [bumper, door, headlight, hood, license plate, logo, mirror, wheel, window, wiper]  
bus [bumper, door, headlight, license plate, logo, mirror, wheel, window, wiper]  
oven [button panel, door, drawer, top]  
cooking stove [burner, button panel, door, drawer, oven, stove]  
microwave [button panel, door, front, side, top, window]  
refrigerator [button panel, door, drawer, side]  
kitchen island [door, drawer, front, side, top]  
dishwasher [button panel, handle, skirt]  
bookcase [door, drawer, front, side]  
television receiver [base, buttons, frame, keys, screen, speaker]  
glass [base, bowl, opening, stem]  
pool table [bed, leg, pocket]  
van [bumper, door, headlight, license plate, logo, mirror, taillight, wheel, window, wiper]  
airplane [door, fuselage, landing gear, propeller, stabilizer, turbine engine, wing]  
truck [bumper, door, headlight, license plate, logo, mirror, wheel, windshield]  
minibike [license plate, mirror, seat, wheel]
Figure B.4: The number of part masks for each object class in ADE20K-Part-234. Each horizontal bar is color-coded to represent a specific part class belonging to the object. The colors of each bar are ordered from bottom to top according to the part sequence in the list of objects with parts.
Figure B.5: Qualitative results on ZSseg+, CATSeg and CLIPSeg concerning the challenging unseen "bird" class in Pascal-Part-116, as shown in the first row. The second row shows the corresponding ground truth. We can observe that CATSeg and CLIPSeg can generalize to the more novel parts: "Bird's Beak" and "Bird's Wing"
washer [button panel, door, front, side]
bench [arm, back, leg, seat]
traffic light [housing, pole]
light [aperture, canopy, diffusor, highlight, light source, shade]
B.3 Data Statistics Analysis.
We report the statistics for the number of object masks that have part annotations in Pascal-Part-116 (see Figure A.1) and ADE20K-Part-234 (see Figure B.3). The total number of part masks for each object and the proportion of each part are shown in Figure B.2 (Pascal-Part-116) and Figure B.4 (ADE20K-Part-234). In Figure B.2, the color sequence from left to right corresponds to the part word sequence as listed in Section B.1. In Figure B.4, the color sequence from bottom to up corresponds to
Figure B.6: Qualitative results on CATSeg's multi-granular generalization ability. From the left to the middle image, the model generalizes from "head" to the more fine-grained "beard". From the middle to the right image, the model generalizes from ["hair", "eyebrow", "eye"] to the coarse-grained "head" and also from "neck" to "torso".
Figure B.7: More qualitative results of generalized zero-shot part segmentation on Pascal-Part-116 are in the first row. The ground truth is in the second row. The seen classes are "cat" and "horse" while the unseen classes are "dog" and "sheep".
the part word sequence as listed in Section B.2. Additionally, we report the scale distribution for the part masks of each object as shown in Figure B.8.
The qualitative results on the comparison among ZSseg+, CATSeg and CLIPSeg for the challenging case "bird" are shown in Figure B.5. Figure B.6 shows the multi-granular generalization ability of the one-stage baselines. The adopted model is CATSeg. The visualization sample is from the "person" class in Pascal-Part-116. We give more qualitative results on Pascal-Part-116 and ADE20K-Part-234 on the three proposed task settings. The adopted model is CLIPSeg with finetuning (VA+L+F+D). The visualization results for the Generalized Zero-Shot Part Segmentation on Pascal-Part-116 and
Figure B.8: The scale ratio (number of pixels in the object part mask out of all pixels in an image.) distribution of all part masks of Pascal-Part-116 (Left) and ADE20K-Part-234 (Right).
Figure B.9: Qualitative results of generalized zero-shot part segmentation on ADE20K-Part-234. The first and second rows show the generalize from the seen classes [chair, armchair, sofa] to the unseen classes [swivel chair, ottoman, stool]. The third row shows the generalize from the seen classes [lamp, chandelier] to the unseen class [fan]. Notably, "fan's blade" is novel at the object and part level.
Figure B.10: Qualitative results of few-shot part segmentation on Pascal-Part-116. We display the segmentation map of four classes: "bird", "aeroplane", "car" and "bicycle".
Figure B.11: Qualitative results of few-shot part segmentation on ADE20K-Part-234. We display the segmentation map of four classes: "lamp", "sink", "toilet" and "cooking stove".