SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The improvement of ST on the unseen category is significant (Pascal:  $81.8\% \rightarrow 86.3\%$ , COCO:  $50.4\% \rightarrow 55.2\%$ ) in terms of  $\mathrm{mIoU}^u$ . However, it's essential to highlight the applicability of ST is limited by a crucial requirement: unseen classes need to be obtained during training. This requirement poses significant limitations on generalizing ST to various scenarios, e.g., open-vocabulary settings, since images of unseen classes may not be obtained during training.
We provide more qualitative results, including typical proposals and top-5  $A^c$  (Fig. 7), as well as examples of models trained on COCO-Stuff and text on A-847 (Fig. 8), A-150 (Fig. 9), PC-459 (Fig. 10), PC-59 (Fig. 11), Pascal-VOC (Fig. 12), and COCO-Stuff(Fig. 13).
Typical Proposals and Top-5  $A^c$ . Fig. 7 shows frozen CLIP and mask-aware CLIP classifications of typical proposals. In the  $2^{nd}$  column, we provide high-quality proposals of thing classes. Both the frozen CLIP and mask-aware CLIP provide high classification scores for the correct classes. In the  $3^{rd}$  column, we provide proposals that only contain part of the objects (row 1-3), and proposals containing more than 1 class (row 4). The mask-aware CLIP provides more proper results compared to the frozen CLIP. In the  $4^{th}$  column, we provide some high-quality background proposals. The frozen CLIP typically gives incorrect predictions, but the mask-aware CLIP assigns high scores for the correct classes.
Qualitative Analysis. Fig. 8,9,10,11,12,13 show segmentation results on Pascal-VOC, COCO-Stuff, ADE20K. In Pascal-VOC dataset (Fig. 12), which only contains 20 thing classes, the FreeSeg + MAFT model tends to assign background regions to the similar thing classes, e.g., "train" in row 1, "pottedplant" in row3-4. "boat" in row 8. In A-847, A-150, PC-459, PC-59 and COCO-Stuff datasets, both seen classes and unseen classes exist in the input images, the FreeSeg + MAFT model generates better segmentation results compared to FreeSeg.
Top5 predictions by mask-aware CLIP
airplane:  $91.6\%$   
waterdrop:  $2.2\%$   
ceiling:  $0.56\%$   
structural:  $0.4\%$   
water:  $0.40\%$
airplane:  $95.7\%$  metal:  $0.78\%$  structural:  $0.7\%$  sky:  $0.29\%$  person:  $0.25\%$
airplane:  $75.6\%$   
kite:  $1.62\%$   
ski:  $1.60\%$   
knife:  $1.59\%$   
structural:  $1.0\%$
airplane:  $18.2\%$  metal:  $13.53\%$  structural:  $6.0\%$  ground:  $2.65\%$  person:  $2.63\%$
airplane:  $95.2\%$   
kite:  $1.01\%$   
sky:  $0.88\%$   
structural:  $0.2\%$   
tie:  $0.20\%$
sky:  $93.22\%$  clouds:  $3.96\%$  ground:  $0.27\%$  airplane:  $0.22\%$  solid:  $0.21\%$
low:  $91.77\%$  horse:  $0.93\%$  sheep:  $0.89\%$  solid:  $0.64\%$  field:  $0.44\%$
cow:  $93.85\%$  food:  $1.06\%$  person:  $0.75\%$  horse:  $0.63\%$  sheep:  $0.62\%$
sheep:  $62.57\%$   
cow:  $34.17\%$   
horse:  $0.45\%$   
vegetable:  $0.19\%$   
person:  $0.17\%$
cow:  $50.62\%$  dog:  $4.98\%$  food:  $4.72\%$  sheep:  $4.07\%$  person:  $3.04\%$
cow:  $66.77\%$  bear:  $5.73\%$  elephant: 3.99 sheep:  $3.25\%$  zebra:  $2.03\%$
sand:  $93.18\%$   
dirt:  $2.97\%$   
sea:  $0.73\%$   
ground:  $0.53\%$   
gravel:  $0.49\%$
Figure 7: Visualizations of typical proposals & top  $5\mathrm{A}^c$  by frozen CLIP and mask-aware CLIP. The correct classes are highlighted in red.
laptop:  $62.11\%$   
desk:  $5.86\%$   
platform:  $4.59\%$   
tv:  $3.81\%$   
cabinet:  $3.72\%$
laptop:  $63.19\%$   
tv:  $10.23\%$   
keyboard:  $3.7\%$   
desk:  $3.30\%$   
playingfield:  $3.0\%$
laptop:  $98.25\%$   
keyboard:  $0.9\%$   
platform:  $0.13\%$   
desk:  $0.09\%$   
mcwave:  $0.04\%$
laptop:  $46.18\%$   
keyboard:  $14.68\%$   
remote:  $3.47\%$   
tv:  $3.22\%$   
desk:  $2.48\%$
board:  $3.61\%$    
rocket:  $3.58\%$    
cloth:  $3.09\%$    
kite:  $2.46\%$    
glove:  $2.17\%$
desk:  $42.38\%$   
diningtable:  $26.3\%$   
table:  $23.77\%$   
furniture:  $1.81\%$   
counter:  $1.74\%$
bottle:  $86.52\%$  water-other:  $1.52\%$  plastic:  $1.20\%$  straw:  $1.04\%$  cup:  $0.87\%$
bottle:  $94.96\%$   
cup:  $0.89\%$   
water:  $0.77\%$   
plastic:  $0.46\%$   
person:  $0.26\%$
hot dog:  $81.14\%$   
food-other:  $7.73\%$   
sandwich:  $3.12\%$   
napkin:  $1.31\%$   
plastic:  $0.93\%$
table:  $35.12\%$   
diningtable:  $27.1\%$   
car:  $12.85\%$   
hot dog:  $12.6\%$   
plastic:  $3.44\%$
water:  $4.83\%$   
wall:  $4.01\%$   
floor:  $3.30\%$   
wall-stone:  $2.51\%$   
solid-other:  $2.49\%$
table:  $61.81\%$  diningtable:  $32.5\%$  furniture:  $0.82\%$  counter:  $0.76\%$  desk-stuff:  $0.52\%$
FreeSeg
Figure 8: Qualitative results on A-847, using 847 class names in ADE20K to generate text embeddings.
FreeSeg+MAFT
FreeSeg
Figure 9: Qualitative results on A-150, using 150 class names in ADE20K to generate text embeddings.
FreeSeg
FreeSeg+MAFT
Figure 10: Qualitative results on PC-459, using 459 class names in Pascal-Context to generate text embeddings.
FreeSeg+MAFT
Figure 11: Qualitative results on PC-59, using 59 class names in Pascal-Context to generate text embeddings.
FreeSeg+MAFT
Figure 12: Qualitative results on Pascal-VOC, using 20 class names in Pascal-VOC to generate text embeddings.
Figure 13: Qualitative results on COCO, using 171 class names in COCO-Stuff to generate text embeddings.