SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We've showcased experimental evidence supporting our method across a diverse set of tasks, including open vocabulary panoptic and semantic segmentation, instance and referring segmentation, and object detection. However, it will be crucial for future work to test our methodology on video-related tasks, such as object tracking and segmentation, to draw comparisons with other universal models like UNNEXT [58]. Furthermore, it's worth considering additional pretraining of our vision encoder on newer, more complex datasets that encompass a vast amount of masks and information. For instance, SA-1B [24], which includes over 1 billion masks, would serve as an ideal training ground. Lastly, it would be advantageous to measure the change in performance when training on supplementary hierarchical datasets. Such an approach will allow us to demonstrate more varied object part segmentations, thereby expanding the capabilities and versatility of our model.
A.7 Broader Impact
Our research introduces a potent approach to hierarchical and universal open vocabulary image segmentation, aiming to address the ever-increasing demand for more data and advanced model architectures. As the demand increases, practical methodologies such as universal segmentation are projected to play a vital role in constructing and training foundational models. Our model, HIPIE, shows promise for fostering progress in a multitude of fields where hierarchical data are critical, including self-driving cars, manufacturing, and medicine. However, it's imperative to acknowledge potential limitations. Given that our model is trained on human annotations and feedback, it can inadvertently replicate any errors or biases present in the datasets. The architecture's complexity is further enhanced when multiple models are integrated, which, in turn, compromises the explainability of the final predictions. Therefore, as with the introduction of any novel technology, it's crucial to implement safety protocols to mitigate misuse. This includes mechanisms for ensuring the accuracy of inputs and establishing procedures to comprehend the criteria the model employs for predictions. By doing so, we can improve the model's reliability and mitigate potential issues.
A.8 Qualitative Results
A.8.1 More Visualizations
We provide more visualizations of panoptic segmentation, part segmentation and referring segmentation in Figs. A2 and A3.
A.8.2 Combining with SAM
We integrate our model with the mask outputs generated by the ViT-H Image encoder from Segment Anything (SAM) [24]. The encoder is trained on SA-1B which encompasses a broad spectrum of objects and masks within each image, enabling us to enhance our segmentation output by utilizing the high-quality masks from the SAM encoder to generate finer, more detailed masks.
To elaborate, in the context of panoptic segmentation, we implement a voting scheme between our pixel-wise annotations and the masks from Segment Anything (SAM), enriching these masks with our labels. For objects where our model demonstrates a strong understanding of hierarchy, such as "person" or "bird", we substitute the SAM masks with ours. This approach enables us to optimize hierarchical outcomes in the face of highly complex images.
Based on our observations from the figures, it's evident that Grounding DINO generates instance segmentation bounding boxes and subsequently uses SAM for the application of the segmentation masks. While this method proves effective for most datasets, SA-1B is a highly complex set featuring a vast array of whole objects, parts and subparts. Our qualitative findings suggest that the a single granularity instance segmentation model may fail to fully capture all objects/parts within an image or may incorrectly identify them. This consequently leads to SAM receiving sub-optimal bounding boxes for segmentation, resulting in fewer and less accurate masks (see third columns in Figs. A4 to A6). In contrast, our methodology (see last columns in Figs. A4 to A6) integrates the SAM encoder masks with our annotations and hierarchical masks wherever feasible. This results in a significantly more fine-grained and accurate output, proving superior in handling complex datasets such as SA-1B.
A.8.3 Combining with Stable Diffusion
As an interesting experiment, we combined our model with image generation model Stable-Diffusion[46] in Fig. A7. Given a source expression and target prompt, we first use HIPIE's segmentation capability to find the corresponding masks, which are then used for image inpainting. Notably, our model can uniquely achieve fine-grained control over object parts by providing part segmentation masks.
Panoptic Segment
Part Segment
Subpart Segment
Referring Segment
Figure A2: More visualizations showcasing panoptic segmentation, part segmentation, subpart segmentation, and referring segmentation results on RefCOCO. It is recommended to view the results in color and zoom in for better detail.
Figure A3: More visualizations showcasing panoptic segmentation, part segmentation, subpart segmentation, and referring segmentation results on RefCOCO. It is recommended to view the results in color and zoom in for better detail.
GDINO+SAM
HIPIE+SAM
Figure A4: Additional results of merging HIPIE with SAM for hierarchical segmentation. By integrating the part masks from our model and conducting a vote among SAM's panoptic masks, we generate finely detailed mask outputs. Our method demonstrates fewer misclassifications and overlooked masks across the SA-1B dataset compared to the Grounding DINO + SAM approach. Furthermore, our technique excels in differentiating between intra-class objects and identifying distinct object parts.
HIPIE+SAM
Figure A5: Additional results of merging HIPIE with SAM for hierarchical segmentation. By integrating the part masks from our model and conducting a vote among SAM's panoptic masks, we generate finely detailed mask outputs. Our method demonstrates fewer misclassifications and overlooked masks across the SA-1B dataset compared to the Grounding DINO + SAM approach. Furthermore, our technique excels in differentiating between intra-class objects and identifying distinct object parts.
HIPIE+SAM
Figure A6: Additional results of merging HIPIE with SAM for hierarchical segmentation. By integrating the part masks from our model and conducting a vote among SAM's panoptic masks, we generate finely detailed mask outputs. Our method demonstrates fewer misclassifications and overlooked masks across the SA-1B dataset compared to the Grounding DINO + SAM approach. Furthermore, our technique excels in differentiating between intra-class objects and identifying distinct object parts.
Generation
Prompt
Figure A7: Results of combining HIPIE with Stable Diffusion for Image inpainting. We leverage our segmentation model to generate masks for the redrawing process. Our model can uniquely achieve fine-grained control by providing part segmentation masks.