Title: NOCTIS: Novel Object Cyclic Threshold based Instance Segmentation
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: novel object instance segmentation, instance segmentation, pretrained vision foundation models, bop challenge, computer vision

Abstract: Instance segmentation of novel objects instances in RGB images, given some example images for each object, is a well known problem in computer vision.
Designing a model general enough to be employed for all kinds of novel objects without (re-) training has proven to be a difficult task.
To handle this, we present a new training-free framework, called: Novel Object Cyclic Threshold based Instance Segmentation (NOCTIS).
NOCTIS integrates two pre-trained models: Grounded-SAM 2 for object proposals with precise bounding boxes and corresponding segmentation masks; and DINOv2 for robust class and patch embeddings, due to its zero-shot capabilities.
Internally, the proposal-object matching is realized by determining an object matching score based on the similarity of the class embeddings and the average maximum similarity of the patch embeddings with a new cyclic thresholding (CT) mechanism that mitigates unstable matches caused by repetitive textures or visually similar patterns.
Beyond CT, NOCTIS introduces: (i) an appearance score that is unaffected by object selection bias; (ii) the usage of the average confidence of the proposals’ bounding box and mask as a scoring component; and (iii) an RGB-only pipeline that performs even better than RGB-D ones.
We empirically show that NOCTIS, without further training/fine tuning, attains state-of-the-art results regarding the mean AP score, w.r.t. the best RGB and RGB-D methods on the seven core datasets of the BOP 2023 challenge for the ''Model-based 2D segmentation of unseen objects'' task.

TL;DR: Our framework outperforms (absolute mean Average Precision) all disclosed models in the BOP 2023 challenge regarding the "Model-based 2D segmentation of unseen objects" task using RGB-only object matching scores.