Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
Authors: Yang Liu
Muzhi Zhu
Hengtao Li
Hao Chen
Xinlong Wang
Chunhua Shen
Venue: ICLR 2024 poster (2024)
Keywords: Vision Foundation Models; Segment Anything; All-Purpose Feature Matching; Training-Free Generalist
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Powered by large-scale pre-training, vision foundation models exhibit significant potential in open-world image understanding. However, unlike large language models that excel at directly tackling various language tasks, vision foundation models require a task-specific model structure followed by fine-tuning on specific tasks. In this work, we present $\textbf{Matcher}$, a novel perception paradigm that utilizes off-the-shelf vision foundation models to address various perception tasks. Matcher can segment anything by using an in-context example without training. Additionally, we design three effective components within the Matcher framework to collaborate with these foundation models and unleash their full potential in diverse perception tasks. Matcher demonstrates impressive generalization performance across various segmentation tasks, all without training. For example, it achieves 52.7% mIoU on COCO-20$^i$ with one example, surpassing the state-of-the-art specialist model by 1.6%. In addition, Matcher achieves 33.0% mIoU on the proposed LVIS-92$^i$ for one-shot semantic segmentation, outperforming the state-of-the-art generalist model by 14.4%. Our visualization results further showcase the open-world generality and flexibility of Matcher when applied to images in the wild.

Published as a conference paper at ICLR 2024
MATCHER: SEGMENT ANYTHING WITH ONE SHOT Us- ING ALL- FEATURE MATCHING
Yang Liu Muzhi Zhu
Hengtao Li
Hao Chen'
Xinlong Wang? Chunhua Shen
1 Zhejiang University, China
(yangliu9610, zhumuzhi, ,iitt haochen cad, chunhuashen) Szjju edu. cn
2 Beijing Academy of Artificial Intelligence vangxinlong@baai. an
ABSTRACT
Powered by large-scale pre-training, vision foundation models exhibit significant potential in open-world image understanding. However, unlike large language mod- els that excel at directly tackling various language tasks, vision foundation models require task-specific model structure followed by fine-tuning on specific tasks. In this work, we present Matcher, a novel perception paradigm that utilizes off-the- shelf vision foundation models address various perception tasks. Matcher can segment anything by using an in-context example without training. Additionally, we design three effective components within the Matcher framework to collaborate with these foundation models and unleash their full potential in diverse perception tasks. Matcher demonstrates impressive generalization performance across various segmentation tasks, all without training. For example, it achieves 52.7% mloU on COCO-20' with one example, surpassing the state-of-the-art specialist model by 1.6%. In addition, Matcher achieves 33.0% mloU on the proposed LVIS-92* for one-shot semantic segmentation, outperforming the state-of-ther the-hhrrtt generalist model by 14.4%. Our visualization results further showease the open-world gener- ality and flexibility of Matcher when applied to images in the wild. Our code is at: https:/ thub. com/ aaaaammmuuo uofa wofa/Mattaie Matcher
INTRODUCTION
Pre-trained on web-scale datasets, large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Chowdhery et al., 2022; Zhang et al., 2022b; Zeng et al., 2022: Touvron et al., 2023), like CharGPT (OpenAI, 2023), have revolutionized natural language processing (NLP). These foundation models (Bommasani al., 2021) show remarkable transfer capability on tasks and data distributions beyond their training scope. LLMs demonstrate powerful zero-shot and few- shot generalization (Brown al., 2020) and solve various language tasks well, e.g., language understanding, generation, interaction, and reasoning.
Research of vision foundation models (VFMs) is catching up with NLP. Driven by auree-scale image- text contrastive pre- training, CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) perform strong zero-shot transfer ability to various classification tasks. DINOv2 (Oquab al., 2023) demonstrates impressive visual feature matching ability by learning to capture complex information at the image and pixel level from raw image data alone. Recently, the Segment Anything Model (SAM) (Kirillov et al., 2023) has achieved impressive class-agnostic segmentation performance by training on the SA-1B dataset, including 1B masks and I images. Unlike LLMs (Brown et al., 2020; Touvron al., 2023), which seamlessly incorporate various language tasks through unified model structure and pre-training method, VFMs face limitations when directly addressing diverse perception tasks. For example, these methods often require a task-specific model structure followed by fine-tuning specific task (He et al., 2022; Oquab et al., 2023).
'Equal contribution. Part of the work was done when YI. was intern at Beijing Academy of Artificial Intelligence. CS is the corresponding author.

---

Published as a conference paper at ICLR 2024
In this work, we aim to find new visual research paradigm: investigating the utilization of VFMs for effectively addressing a wide range of perception tasks, e.g., semantic segmentation, part segmentation, and video object segmentation, without training. Using foundation models is non-trivial due to the following challenges: 1) Although VFMs contain rich knowledge, it remains challenging to directly leverage individual models for downstream perception tasks. Take SAM an example. While SAM can perform impressive zero-shot class-agnostic segmentation performance across various tasks, it cannot provide the semantic categories for the predicted masks. Besides, SAM prefers to predict multiple ambiguous mask outputs. It is difficult to select the appropriate mask as the final result for different tasks. 2) Various tasks involve complex and diverse perception requirements. For example, semantic segmentation predicts pixels with the same semantics. However, video object segmentation needs to distinguish individual instances within those semantic categories. Additionally, the structural distinctions of different tasks need to be considered, encompassing diverse semantic granularities ranging from individual parts to complete entities and multiple instances. Thus, naively combining the foundation models can lead to subpar performance.
To address these challenges, we present Matcher, a novel perception framework that effectively incorporates different foundation models for tackling diverse perception tasks by using single in-context example. We draw inspiration from the remarkable generalization capabilities exhibited by LLMs in various NLP tasks through in-context learning (Brown al., 2020). Prompted by the in-context example, Matcher can understand the specific task and utilizes DINOV2 to locate the target by matching the corresponding semantic feature. Subsequently, leveraging this coarse location information, Matcher employs SAM to predict accurate perceptual results. In addition, we design three effective components within the Matcher framework collaborate with foundation models and fully unleash their potential in diverse perception tasks. First, we devise bidirectional matching strategy for accurate cross- cross-image semantic dense matching and a robust prompt sampler for mask proposal generation. This strategy increases the diversity of mask proposals and suppresses fragmented false-positive masks induced by matching outliers. Furthermore, we perform instance- level matching between the reference mask and mask proposals to select high-quality masks. We utilize three effective metrics, i.e., emd, purity, and coverage, to estimate the mask proposals based on semantic similarity and the quality of the mask proposals, respectively. Finally, by controlling the number of merged masks, Matcher can produce controllable mask output to instances of the same semantics in the target image.
Our comprehensive experiments demonstrate that Matcher has superior generalization performance across various segmentation tasks, all without the need for training. For one-shot semantic seg- mentation, Matcher achieves 52.7% mIoU on COCO-20' (Nguyen & Todorovic, 2019), surpassing the state-of-the-art specialist model by 1.6%, and achieves 33.0% mloU on the proposed LVIS- 92', outperforming the state-of-the-ar generalist model SegGPT (Wang et al., 2023b) by 14.4%. And Matcher outperforms concurrent PerSAM (Zhang et al., 2023) by large margin (+29.2% mean mloU on COCO-20', +11.4% mloU on FSS-1000 (Li et al., 2020), and + +10.7 % mean mIoU on LVIS-92'), suggesting that depending solely on SAM limits the generalization capabilities for semantically-driver tasks, e.g., semantic segmentation. Moreover, evaluated on two proposed benchmarks, Matcher shows outstanding generalization on one-shot object part segmentation tasks. Specifically, Matcher outperforms other methods by about 10.0% mean mloU on both benchmarks. Matcher also achieves competitive performance for video object segmentation on both DAVIS 2017 val (Pont-Tuset et al., 2017) and DAVIS 2016 val (Perazzi al., 2016). In addition, exhaustive abla- tion studies verify the effectiveness of the proposed components of Matcher. Finally, our visualization results show robust generality and flexibility never seen before.
Our main contributions are summarized as follows:
We present Matcher, one of the first perception frameworks for exploring the potential of vision foundation models in tackling diverse perception tasks, e.g. one-shot semantic segmentation, one-shot object part segmentation, and video object segmentation.
We design three components, i.e., bidirectional matching, robust prompt sampler, and instance-level matching, which can effectively unleash the ability of vision foundation models to improve both the segmentation quality and open-set generality.
Our comprehensive results demonstrate the impressive performance and powerful gener- alization of Matcher. Sufficient ablation studies show the effectiveness of the proposed components.
2

---

Published as a conference paper at ICLR 2024
Correspondence Matrix Extraction
Prompts Generation
#--
Patch-Level Matching (PLM)
Robust Prompt Sampler (RPS)
Image Encoder
Correspondence Matrix
Thiget
M Mask Proposals
Controllable Masks Merging
Instance-Lewe Level Matching (LM)
Promptable Segmenter
(CMM)
Target Mook
Selected Masks
Controllable Masks Generation
Prompts
Figure I: An overview of Matcher. Our training- free framework addresses various segmentation tasks through three operations: Correspondence Matrix Extraction, Prompts Generation, and Controllable Masks Generation.
RELATED WORK
Vision Foundation Models Powered by large-scale pre-training, vision foundation models have achieved great success in computer vision. Motivated by masked language modeling (Devlin et al., 2019; Liu et al., 2019) in natural language processing, MAE (He et al., 2022) uses asymmetric encoder-decoder and conducts masked image modeling to effectively and efficiently train scalable vision Transformer (Dosovitskiy et al., 2020) models. CLIP (Radford et al., 2021) learns image representations from scratch on 400 million image-text pairs and demonstrates impressive zero-shot image classification ability. By performing image and patch level discriminative self supervuee learning, DINOv2 (Oquab et al., 2023) learns all-purpose visual features for various downstream tasks. Recently, pre-trained with 1B masks and 11M images, Segment Anything Model (SAM) (Kirillov et al., 2023) emerges with impressive zero-shot class-agnostic segmentation performance. Although vision foundation models have shown exceptional fine-tuning performance, they have limited ca- pabilities in various visual perception tasks. However, large language models (Brown et al., 2020; Chowdhery et al., 2022: Touvron al., 2023), like ChatGPT (OpenAI, 2023), can solve a wide range of language tasks without training. Motivated by this, this work shows that various perception tasks can be solved training-free by utilizing off-the-shelf vision foundation models to perform in-context inference.
Vision Generalist for Segmentation Recently, a growing effort has been made unify various segmentation tasks under single model using Transformer architecture (Vaswani al., 2017). The generalist Painter (Wang et al., 2023a) redefines the output different vision tasks as images and utilizes masked image modeling continuous pixels to perform in in-context training with supervised datasets. As a variant of Painter, SegGPT (Wang et al., 2023b) introduces a novel random coloring approach for in-context training improve the model's generalization ability. By prompting spatial queries, egg., points, and text queries, egg., textual prompts, SEEM (Zou et al., 2023) performs various segmentation tasks effectively. More recently, PerSAM and PerSAM-F (Zhang et al., 2023) adapt SAM for personalized segmentation and video object segmentation without training or with two trainable parameters. This work presents Matcher, training-free framework for segmenting anything with one shot. Unlike these methods, Matcher demonstrates impressive generalization performance across various segmentation tasks by integrating different foundation models.
METHOD
Matcher a training-fres framework that segments anything with one shot by integrating an all- purpose feature extraction model (e.g.. DINOv2 (Oquab et al., 2nd class-agnostic segmenta- tion model (e.g., SAM (Kirillov et al., 2023)). For the given in-context example, including reference image x, and mask m,, Matcher same semantics. The overview Matcher depicted in Fig. 1. Our framework consists of three components: Correspondence Matrix Extraction (CME), Prompts Generation (PG), and Controllable
segment the objects or parts of target image X, with the
3

---

Published as a conference paper at ICLR 2024
Step forward matching
P= (pt )4
= sim(Pr, 2+)
Pt (D:)
reference
target
1==1 Hungarian
Step 3. mask filtering
Step 2. reverse matching
S sim(a,, Hungarian
P. [p-
P7 (p()
P= Pp
Illustration of the proposed bidirectional matching. Bidirectional matching consists of three steps: forward matching, reverse matching, and mask filtering. Purple points denote the matched points. Red points denote the outliers.
Figure 2:
Masks Generation (CMG). First, Matcher extracts correspondence matrix by calculating the simi- larity between the image features of x, and x, Then, we conduct patch-level matching, followed by sampling multiple groups of prompts from the matched points. These prompts serve as inputs to SAM, enabling the generation of mask proposals. Finally, we perform an instance-level matching between the reference mask and mask proposals select high- nigh-quality masks. We elaborate on the three components in the following subsections.
3.1 CORRESPONDENCE MATRIX EXTRACTION
We rely on off-the-self image encoders to extract features for both the reference and target images. Given inputs x, and Xx, the encoder outputs patch-level features Zr, Z, E Hx Patch-wise similarity between the two features is computed to discovery the best matching regions of the reference mask on the target image. We define a correspondence matrix g. XHW as follows,
(S)
z
where (S), denotes the cosine similarity between i-th patch feature zi of z, and j-th patch feature z! of z, We can denote the above formulation in compact form as S = sim 2,, 2t).
Ideally, the matched patches should have the highest similarity. This could be challenging in practice, since the reference and target objects could have different appearances or even belong to different categories. This requires the encoder to embed rich and detailed information in these features.
3.2 PROMPTS GENERATION
Given the dense correspondence matrix, can get coarse segmentation mask by selecting the most similar patches in the target image. However, this naive approach leads to inaccurate, fragmented result with many outliers. Hence, we use the correspondence feature generate high quality point and box guidance for promptable segmentation. The process involves bidirectional patch matching and diverse prompt sampler.
Patch-I Level Matching The encoder tends to produce wrong matches in hard cases such ambiguous context and multiple instances. We propose a bidirectional matching strategy to eliminate the matching outliers.
As shown in Fig. 2, we first perform bipartite matching between the points on the reference mask P. {p/). 1h and z, obtain the forward matched points on the target image
pr+ {pit using the forward correspondence matrix S
Then, perform another bipartite matching, named the reverse matching between P. and z, to obtain the reverse matched points on the reference image p. { using the reverse correspondence matrix S sim(z,; P,7).
4

---

Published as a conference paper at ICLR 2024
Finally, we filter out the points in the forward set if the corresponding reverse points not the reference mask my- The final matched points are P {p) Pr in my).
Robust Prompt Sampler Inspired by the effective prompt-engineering (Kojima et al., 2022; Wei et al., 2022; Li & Liang, 2021; Zhu et al., 2023), introduce robust prompt sampler for the promptable segmenter to support robust segmentation with various semantic granularity, from parts and whole to multiple instances. We first cluster the matched points P based on their locations into K' clusters PR with k-means++ (Arthur & Vassilvitskii, 2007). Then the following three types of subsets are sampled as prompts:
Part-level prompts sampled within each cluster P' C Pk:
Instance-level prompts are sampled within all matched points p C P:
Global prompts are sampled within the set of cluster centers P9 C C' to encourage coverage, where C {c1, .,,..,, } are the cluster centers.
In practice, we find this strategy not only increases the diversity of mask proposals but also suppresses fragmented false-positivy masks induced by matching outliers.
3.3 CONTROLLABLE MASKS GENERATION
The edge features of an object extracted by the image encoder can confuse background information, inducing some indistinguishable outliers. These outliers can generate some false-positive masks. To overcome this difficulty, further select high-quality masks from the mask proposals via an instance-level matching module and then merge the selected masks to obtain the final target mask.
Instance-Level Matching We perform the instance-level matching between the reference mask and mask proposals to select great masks. We formulate the matching to the Optimal Transport (OT) problem and employ the Earth Mover's Distance (EMD) compute structural distance between dense semantic features inside the masks to determine mask relevance. The cost matrix of the OT problem can be calculated by C } S). We the method proposed in (Bonneel al., 2011) to calculate the EMD, noted as emd.
In addition, we propose two other mask proposal metrics, i.., purity Num((pmp) and coverage Arca(mp)
Num((mpp), to assess the quality of the mask proposals simultaneously, where P {p} E Num(P)
Ip' in mp), Num(() represents the number of points, Area(-) represents the area of the mask, and mp is the mask proposal. A higher degree of purity promotes the selection of part-level masks, while a higher degree of coverage promotes the selection of instance-level masks. The false-positive mask fragments can be filtered using the proposed metries through appropriate thresholds, followed by a score-based selection process identify the top-k highest-quality masks
score = a -1-emm) + B purity coverage*
(2)
where a, B, and are regulation coefficients between different metrics. By manipulating the number of merged masks, Matcher can produce controllable mask output to instances of the same semantics in the target image. More details of emd, purity and coverage are provided in Appendix
EXPERIMENTS
4.1 EXPERIMENTS SETTING
Vision Foundation Models We use DINOv2 (Oquab et al., 2023) with a ViT-L/14 (Dosovitskiy al., 2020) as the default image encoder of Matcher. Benefiting from large-scale discriminative self-supervised learning at both the image and patch level, DINOv2 has impressive patch-level representation ability, which promotes exact patch matching between different images. We use the Segment Anything Model (SAM) (Kirillov et al., 2023) with ViT-H as the segmenter of Matcher. Pre-trained with 1B masks and 1M images, SAM emerges with impressive zero-shot segmentation performance. Combining these vision foundation models has the enormous potential to touch open- world image understanding, In all experiments, we do not perform any training for the Matcher. More implementation details are provided in Appendix B.
5

---

Published as a conference paper at ICLR 2024
COCO-20'
FSS-1000
LVIS-92
Methods
Venue
one-shot few-shot
one-shot few-shot one-shot few-shot
specialist model
HSNet (Min et al., 2021)
ICCV'21
41.2
49.5
86.5
88.5
17.4
22.9
VAT (Hong et al., 2022)
ECCV'22
41.3
47.9
90.3
90.8
18.5
22.7
FPTrans (Zhang et al., 2022a)
2eur1PS'22
47.0
58.9
MSANet (Iqhal et al., 2022)
arXiv'22
51.1
56.8
generalist model
Painter (Wang et al., 2023a)
CVPR'23
33.1
32.6
61.7
62.3
10.5
10.9
SegGPT (Wang et al., 2023b)
ICCV'23
56.1
67.9
85.6
89.3
18.6
25.4
PerSAM11 (Zhang et al., 2023)
23.0
71.2
11.5
PerSAM-F arXiv'23
23.5
75.6
12.3
Matcher
this work
52.7
60.7
87.0
89.6
33.0
40.0
Table 1: Results of few-shot semantic segmentation on COCO-20', FSS-1000, and LVIS-92', Gray indicates the model is trained by in-domain datasets. f indicates the training-free frreing--reeeee method. indicates the method using SAM. Note that the training data of SegGPT includes COCO.
4.2 FEW-SHOT SEMANTIC SEGMENTATION
Datasets For fho-shot semantic segmentation, we evaluate the performance of Matcher on COCO- 20 (Nguyen& & Todorovic, 2019), SSS-1000 (Li et al., 2020), and LVIS-92'. COCO-20' partitions the 80 categories of the MSCOCO dataset (Lin al., 2014) into four cross-validation folds, each containing 60 training classes and 20 test classes. FSS-1000 consists of mask-annotated images from 1,000 classes, with 520, 240, and 240 classes in the training, validation, and test sets, respectively. We verify Matcher the test sets of COCO-20' and FSS-1000 following the evaluation scheme of (Min et al., 2021). Note that, different from specialist models, we do not train Matcher on these datasets. In addition, based the LVIS dataset (Gupta et al., 2019), we create LVIS-92', more challenging benchmark for evaluating the generalization of model across datasets. After removing the classes with less than two images, we retained total of 920 classes for further analysis. These classes were then divided into 10 equal folds for testing purposes. For each fold, we randomly sample a reference image and target image for evaluation and conduct 2,300 episodes.
Results We compare the Matcher against a variety of specialist models, such as HSNet (Min et al., 2021), VAT (Hong et al., 2022), FPTrans (Zhang et al., 2022a), and MSANet (Iqbal et al., 2022), as well as generalist models like Painter (Wang et al., 2023a), SegGPT (Wang et al., 2023b), and PerSAM (Zhang et al., 2023). As shown in Table 1, for COCO-20' Matcher achieves 52.7% and 60.7% mean mloU with one-shot and few-shot, surpassing the state-of-the-an specialist models MSANet and achieving comparable with SegGPT. Note that the training data SegGPT includes COCO. For FSS-1000 Matcher exhibits highly competitive performance compared with specialist models and surpasses all generalist models. Furthermore, Matcher outperforms training- fraining-free free PerSAM and fine-tuning PerSAM-F by significant margin (+29.2% % mean mloU on COCO-20', +11.4% mIoU on 1000, and +10.7% % mcan mIoU on LVIS-92'), (2-), suggesting that depending solely on SAM results in limited generalization capabilities for semantic tasks. For LVIS-92', we compare the cross-dataset generalization abilities of Matcher and other models. For specialist models, we report the average performance of four pre-trained models on COCO-20', Matcher achieves 33.0% % and 40.0% mean mIoU with one-shot and few-shot, outperforming the state-of-the-art generalist model SegGPT by 14.4% % and 14.6% %. Our results indicate that Matcher exhibits robust generalization capabilities that are not present in the other models.
4.3 ONE-SHOT OBJECT PART SEGMENTATION
Datasets Requiring a fine-grained understanding of objects, object part segmentation is a more challenging task than segmenting an object. We build two benchmarks to evaluate the performance of Matcher on one-shot part segmentation, i.e., PASCAL-Part and PACO-Part, Based on PASCAL VOC 2010 (Everingham et al., 2010) and its body part annotations (Chen et al., 2014), we build the PASCAL-Part dataset following (Morabia et al., 2020). The dataset consists of four superclasses, iee, animals, indoor, person, and vehicles. There are five subclasses for animals, three for indoor, one for person, and six for vehicles. There are 56 different object parts in total. PACO (Ramanathan et al., 2023) is a newly released dataset that provides 75 object categories and 456 object part categories. Based on the PACO dataset, we build the more difficult PACO-Part benchmark for one-shot object part segmentation. We filter the object parts whose area is minimal and those with less than two images, resulting in 303 remaining object parts. We split these parts into four folds, each with about

---

Published as a conference paper at ICLR 2024
PASCAL-Pais
PACO-Part
Methods Venue
animals
indoor person vehicles mean
F2
F3
mean
HSNet (Min et al., 2021) ICCV'21
21.2
53.0
20.2
35.1
32,4
20.8 21.3
25,5
22.6
22.6
VAT (Hong al., 2022) ECCV'22
21.5
55.9
20.7
36.1
33.6
22.0
22.9
26.0
23.1
23.5
Painter (Wang et al., 2023a) CVPR'23
20.2
49.5
17.6
34.4
30.4
13.7
12.5
15.0
15.1
14.1
SegGPT (Wang et al., 2023b) ICCV'23
22.8
50.9
31.3
38.0
35.8
13.9
12.6 14.8
12.7
13.5
PerSAM (Zhang al., 2023) arXiv'23 23
19.9
51.8
18.6
32.0
30.1
19.4
20.5
23.8
21.2
21.2
Matcher this work
37.1
56.3
32.4
45.7
42.9
32.7
35.6
36.5 34.1
34.7
Table 2: Results of one-shot part segmentation on PASCAL-Part and PACO-Part. i indicates the
training-free method. $ indicates the method using SAM.
DAVIS 2017 val J&F
DAVIS 2016 val
Methods
Venue
J
F
J&F
J
F
with video data
AGAME (Johnander et al., 2019)
CVPR'19 19
70.0
67.2
72.7
AGSS (Lin et al., 2019)
ICCV'19
67.4
64.9
69.9
AFB-URR (Liang et al., 2020)
NeurIPS'20 20
74.6
73.0
76.1
AOT (Yang et al., 2021)
NeurlPS'21
85.4
82.4
88.4
92.0
90.7
93.3
SWEM (Lin et al., 2022)
CVPR'22
84.3
81.2
87.4
91.3
89.9
92.6
XMem (Cheng & Schwing. 2022)
ECCV'22
87.7
84.0
91.4
92.0
90.7
93.2
without video data
Painter ( (Wang et al., 2023a)
CVPR'23
34.6
28.5
40.8
70.3
69.6
70.9
SegGPT (Wang ct al., 2023h)
ICCV'23
75.6
72.5
78.6
83.7
83.6
83.8
PerSAM (Zhang et al., 2023)
60.3
56.6
63.9
arXiv'23
PerSAM-F
71.9
69.0
74.8
Matcher
this work
79.5
76.5
82.6
86.1
85.2
86.7
Table 3: Results of video object segmentation on DAVIS 2017 val, and DAVIS 2016 val. Gray indicates the model is trained on target datasets with video data. indicates the training-free method. $ indicates the method using SAM.
76 different object parts, We crop all objects out with their bounding box to evaluate the one-shot part segmentation on both two datasets. More details are provided in Appendix
Results We compare our Matcher with HSNet, VAT, Painter, and PerSAM. For HSNet and VAT, we use the models pre-trained on PASCAL-5' (Shaban et al., 2017) and COCO-20 20y for PASCAL-Part and PACO-Part, respectively. As shown in Table 2, the results demonstrate that Matcher outperforms all previous methods by large margin. Specifically, Matcher outperforms the SAM-based PerSAM + 12.8% mean mloU on PASCAL-Part and + 13.5% on PACO-Part, respectively. SAM has shown the potential to segment any object into three levels: whole, part, and subpart (Kirillov et al., 2023). However, it cannot distinguish these ambiguity masks due to the lack of semantics. This suggests that SAM alone cannot work well on one-shot object part segmentation. Our method empowers SAM for semantic tasks by combining with an all-purpose feature extractor and achieves effective generalization performance on fine-grat -re-gra object part segmentation tasks with an in in-context example.
4.4 VIDEO OBJECT SEGMENTATION
Datasets Video object segmentation (VOS) aims to segment a specific object in video frames. Following Wang et al. (2023b), we evaluate Matcher on the validation split of two datasets, i.e., DAVIS 2017 val (Pont-Tuset et al., 2017), and DAVIS 2016 val (Perazzi et al., 2016), under the semi-supervised VOS setting. Two commonly used metrics in VOS, the J score and the F score, are used for evaluation.
Details In order to track particular moving objects in video, maintain reference memory containing features and the intermediate predictions of the previous frames in Matcher. We determine which frame retain in the memory according to the score (see subsection 3.3) of the frames. Considering that objects are more likely to be similar to those in adjacent frames, we apply a decay ratio decreasing by time to the score. We fix the given reference image and mask in the memory to avoid failing when some objects disappear in intermediate frames and reappear later.
Results We compare Matcher with the models trained with without video data different datasets in Table 3. The results show that Matcher can achieve competitive performance compared with the models trained with video data. Moreover, Matcher outperforms the models trained without video data, e.g., SegGPT and PerSAM-F, on both two datasets. These results suggest that Matcher can effectively generalize to VOS tasks without training.
7

---

Published as a conference paper at ICLR 2024
COCO-20" mean mloU
FSS-1000) mloU
DAVIS 2017
Strategy
COCO-20'
FSS- SS0-1000
DAVIS 2017
T.M
J&FF
mean mloU
mloU
J&F
29.0
76.2
39.9
forward
50.6
81.1
73.5
reverse
21.4
47.7
41.3
52.7
87.0
79.5
bidirectional
52.7
87.0
79.5
(a) Ablation study ILM.
(h) Ablation study of bidirectional matching.
emd
p&cc
mean mloU
COCO-20
FSS- 1000
mIoU
DAVIS 2017
J&F
Frames
DAVIS 2017
51.3
86.3
67.5
J&F J
73.5
74,4
79.5
78.0
35.3
86.3
76.3
70.0
70.5
74.9
52.7
87.0
79.5
F
77.5
78.2
82.6
81.1
(c) Effect of different mask proposal metrics.
(d) Effect of the number of frames for VOS.
Table 4: Ablation study. We report the mean mloU of four folds on COCO-20', mloU on FSS-1000, and J& F on DAVIS 2017 val. Default setting settings are marked in Gray
Marcher
SpoOPT
ParSAMFF
(a) Object and object part segmentatiinn.
Cross-style object and segmentation.
reforence
GT
(c) Controllable mask oulput.
Figure 3: Qualitative results one-shot segmentation.
4.5 ABLATION STUDY
As shown Table 4, we conduct ablation studies on both the difficult COCO-20' dataset and the simple FSS-1000 dataset for one-shot semantic segmentation and DAVIS 2017 val for video object segmentation to sufficiently verify the effectiveness of our proposed components. In this subsection, explore the effects of matching modules (ILM), patch-level matching strategies, and different mask proposal metrics.
Ablation Study of ILM Patch-level matching (PLM) and instance-level matching (ILM) are the vital components of Matcher that bridge the gap between the image encoder and SAM to solve various few-shot perception tasks training-free. As shown in Table 4a, PLM builds the connection between matching and segmenting and empowers Matcher with the capability of performing various few-shot perception tasks faining-free, And ILM enhances this capability by large margin.
Ablation Study of Bidirectional Matching As shown in Table 4b, we explore the effects of the forward matching and the reverse matching of the proposed bidirectional matching. For the reverse matching, because the matched points P. (see subsection 3.2) are unavailable when performing reverse matching directly, we perform the reverse matching between Z, and z,.. Without the guidance of the reference mask, reverse matching (line 2) produces many wrong matching results, resulting in poor performance. Compared with the forward matching (line 1), our bidirectional matching strategy improves the performance by +2. 1% mean mloU COCO-20', by +5.9% mloU on FSS- 1000, and by +6.0% J&F on DAVIS 2017. These significant improvements show the effectiveness of the proposed bidirectional matching strategy.
8

---

Published as a conference paper at ICLR 2024
SegGPy
Pur44M-
SepOPT
PaGAMMF
Our
Figure 4: Qualitative results of video object segmentation on DAVIS 2017.
Ablation Study of Different Mask Proposal Metrics As shown in Table 4c, emd is more effective on the complex COCO-20' dataset. emd evaluates the patch-level feature similarity between the mask proposals and the reference mask that encourages matching all mask proposals with the same category. In contrast, by using purity and coverage, Matcher can achieve great performance on DAVIS 2017. Compared with emd, purity and coverage are introduced to encourage selecting high-quality mask proposals. Combining these metrics estimate mask proposals, Matcher can achieve better performance in various segmentation tasks without training.
Effect of the Number of Frames for VOS As shown in Table 4d, also explore the effect of the number of frames on DAVIS 2017 val. The performance of Matcher can be improved as the number of frames increases, and the optimal performance is achieved when using four frames. More ablation studies are provided in Appendix D.
4.6 QUALITATIVE RESULTS
To demonstrate the generalization of our Matcher, we visualize the qualitative results of one-shot segmentation in Fig. 3 from three views, iee,, object and object part segmentation, cross-style object and object part segmentation, and controllable mask output. Our Matcher can achieve higher-quality objects and parts masks than SegGPT and PerSAM-F. Better results cross-style segmentation show the impressive generalization of Matcher due to effective all-feature matching. In addition, by manipulating the number of merged masks, Macther supports multiple instances with the same semantics. Fig. 4 shows qualitative results of VOS on DAVIS 2017. The remarkable results demonstrate that Matcher can effectively unleash the ability foundation models to improve both the segmentation quality and open-set generality.
CONCLUSION
In this paper, we present Matcher, training-free framework integrating off-the-shelf vision foundation models for solving various few-shot segmentation tasks. Combining these foundation models properly leads to positive synergies, and Matcher emerges complex capabilities beyond individual models. The introduced universal components, i.e., bidirectional matching, robust prompt sampler, and instance- level matching, can effectively unleash the ability of these foundation models. Our experiments demonstrate the powerful performance of Matcher for various few-shot segmentation tasks, and our visualization results show open- open-world world generality and flexibility on images the wild.
Limitation and Ethics Statement While Matcher demonstrates impressive performance for semantic- level segmentation, e.g., one-shot semantic segmentation and one-shot object part segmentation, has relatively limited instance-level matching inherited from the image encoder, which restrains its performance for instance segmentation. However, the comparable VOS performance and the visualization of controllable mask output demonstrate that Matcher has the potential for instance-level segmentation. We will explore it in future work. Our work can unleash the potential of different foundation models for various visual tasks. In addition, our Matcher is built upon open-source foundation models without training, significantly reducing carbon emissions. We not foresee any obvious undesirable ethical social impacts now.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Science and Technology Major Project (No. 11222D011800/ The research was in part supported by the Supercomputing Center of Hangzhou City University, which provided advanced computing resources.
REFERENCES
David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proc. Ann. ACM SIAM Symp. on Disc. Algo., 2007.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. Displacement interpolation using lagrangian mass transport. In Proc. of the SIGGRAPH Asia conf., 2011.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In Proc. Adv. Neural Inf. Process. Syst. 2020.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In IEEE Conf. Comput. Vis. Pattern Recog., 2014.
Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson-shifie memory model. In Eur. Conf. Comput. Vis., 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint Sttiv:2204.023111 2022.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Nor. Amer. Chap. of the ACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. Learn. Represent.,
2020.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. Int. J. Comput. Vis., 2010.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In IEEE Conf. Comput. Vis. Pattern Recog. 2022.
Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. Eur. Conf. Comput. Vis., 2022.
Ehtesham Iqbal, Sirojbek Safarov, and Seongdeok Bang. Msanet: Multi-similarity and attention guidance for boosting few-shot segmentation. arXiv preprint arXiv:2206.0996 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and languae representation learning with noisy text supervision. In Proc. Inf. Conf. Mach. Learn., 2021.
Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg. generative appearance model for end-to-end video object segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.. 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint S.Hi::2300.02644.. 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proc. Adv. Neural Inf. Process. Syst., 2022.
10

---

Published as a conference paper at ICLR 2024
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic- sam: Segment and recognize anything at any granularity. arXiv preprint XXXv:2237.04767, 2023.
Xiang Li, Tianhan Wei, Yau Pun Chen, Yu- Wing Tai, and Chi-Keung Tang. Fso0 A 1000-class dataset for few-shot segmentation. In IEEE Conf. Comput. Vis. Puttern Recog., 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 111:2200.001190 2021.
Yongging Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement. In Proc. Adv. Neural Inf. Process. Syst., 2020.
Huaijia Lin, Xiaojuan Qi, and Jiaya Jia. Agss-vos: Attention guided single-shot video object segmentation. In Int. Conf. Comput. Vis., 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., 2014.
Zhihui Lin, Tianyu Yang, Maomao Li, Ziyu Wang, Chun Yuan, Wenhao Jiang, and Wei Liu. Swem: Towards real-time video object segmentation with sequential weighted iopectation-maximizaiin In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 11692, 2019.
Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.. 2021.
Keval Morabia, Jatin Arora, and Tara Vijaykumar. Attention- baatid joint detection of object and semantic part. arXiv preprint Sxkiv:2007.024111 2020.
Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Int. Conf. Comput. Vis., 2019.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 110v::2344.0799333 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proc. Adv. Neural Inf. Process. Syst., 2022.
Federico Perazzi, Jordi Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Horuung. A benchmark dataset and evaluation methodology for video object segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2016.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint
10X:01704.00675. 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn., 2021.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. arXiv preprint arXiv:2301.01770
2023.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In Brit. Mach. Vis. Conf', 2017.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Adv. Neural Inf. Process. Syst., 2017.
11

---

Published as a conference paper at ICLR 2024
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. IEEE Conf. Comput. Vis. Pattern Recog., 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. In Ini. Conf. Comput. Vis., 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning large language models. In Proc. Adv.
Neural Inf. Process. Syst. 2022.
Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In Proc. Adv. Neural Inf. Process. Syst., 2021.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.0244 10.02444, 2022.
Jian-Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxi transformer for few-shot segmentation. In Proc. Adv. Neural Inf. Process. Syst., 2022a.
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.085: 033044 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trate pre-trained transformer language models. arXiv preprint arXiv:2205.0 00000, 2022b.
Muzhi Zhu, Hengtao Li, Hao Chen, Chengxiang Fan, Weian Mao, Chenchen Jing. Yifan Liu, and Chunhua Shen. Segprompt: Boosting open-world segmentation via category-level prompt learning. In Int. Conf. Comput. Vis., 2023.
Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee, Segment everything everywhere all at once. arXiv preprint arXiv:2304.06711 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A MORE DETAILS OF INSTANCE-LEVEL MATCHING
The emd metric. The OT problem can be described as follows: suppose that m suppliers U {u.li 1,2,..., m require transport goods for demanders D {dg 1, 1.2..., where represents supply units i-th supplier and d, denotes
demand of j-th demanded. The cost of transporting each unit of goods from the i-th supplier to the j-th demander is represented by Cj+ and the number of units transported is denoted by Tij. The goal of the OT problem to identify a transportation plan
(mylli 1, m,, ....n that minimizes the overall transportation cost
min
Tij
i=1 j=1
m
s.t.
j=1 Tij = Ui
Tij dj. j=1
:=1
(3)
m
U1 Σ
i=1
Tij 2 0, i= 1, m. j 1,, ....
In the context of Matcher, the suppliers and the demanders are n target image patches covered by the mask proposal (produced by SAM). The goods that the suppliers need to transmit have the same value, i.e., u, m > t4 = 1. Similarly, n Ed = The cost est
reference image patches covered by the reference mask,
the goods that the demanders need also have the same value, i.e. d, 1 can be obtained from the cost matrix C by utilizing the mask proposal mp and the reference mask my. Then, we use the method proposed in Bonneel et al. (2011) to calculate the EMD.
coverage
coverage
purity
purity
(a) Single instance
(b) Multiple instances
Figure 5: Illustration the effects of the purity and coverage.
The purity and coverage metries Fig. shows examples demonstrate the effects of the purity and coverage criteria in two scenarios, i.e., single instance and multiple instances. A higher degree of purity promotes the selection of part or single instance masks, while a higher degree coverage promotes the selection of whole multiple instance masks.
B IMPLEMENTATION DETAILS
We DINOv2 (Oquab al., 2023) with ViT-L/14 (Dosovitskiy et al., 2020) as the default image encoder of Matcher. And we use the Segment Anything Model (SAM) (Kirillov et al., 2023) with ViT-H as the segmenter Matcher. Matcher. We set input image sizes are 518 x
all experiments, we do not perform any training for the
518 for one-shot semantic segmentation and object part segmentation and 896 x 504 for video object segmentation. We conduct experiments from three semantic granularity for semantic segmentation, i.e., parts (PASCAL-Parr and PACO-Part). whole (FSS-1000), and multiple instances (COCO-20' and LVIS-92'). We set the number of clusters to 8. For COCO-20* and LVIS-92 we sample the instance-level points from the matched points and dense image points to encourage SAM to output more instance masks. We set the filtering thresholds emd and purity to 0.67, 0.02 and set a, / and À to 1.0, 0.0, and 0.0, respectively. For FSS-1000, sample the global prompts from centers. We set a, 3, and to 0.8, 0.2, and 1.0, respectively. We sample the points from the matched points and use the smallest axis-aligned box containing these
13

---

Published as a conference paper at ICLR 2024
matched points for PASCAL-Part and PACO-Part. We set the filtering threshold coverage to 0.3 and set a, 3 and to 0.5, 0.5, and 0.0, respectively. For video object segmentation, we sample the global prompts from centers. We set the filtering threshold emd to 0.75 and set a, S, and À to 0.4, 1.0, and 1.0.
C DATASET DETAILS
PASCAL-Part Based on PASCAL VOC 2010 (Everingham al., 2010) and its body part annota- tions (Chen et al., 2014), we build the PASCAL-Part dataset following (Morabia et al., 2020). Table shows the part taxonomy of PASCAL-Part dataset. The dataset consists of four superclasses, i.e., animals, indoor, person, and vehicles. There are five subclasses for animals (bird, cat, cow, dog, horse, sheep), three for indoor (bottle, potted plant, tv monitor), one for person (person), and six for vehicles (acroplane, bicycle, bus, car, motorbike, train). There are 56 different object parts in total.
PACO-Part Based on the PACO (Ramanathan et al., 2023) dataset, we build the more difficult PACO-Part benchmark for one-shot object part segmentation. Firstly, filter the categories having only sample. Then, we filter low-quality examples with an extremely small pixel area within PACO, which leads to significant noise during evaluation, resulting in 303 remaining object parts. Table 6 shows the part taxonomy the PACO-Part dataset. We split these parts into four folds, each with about 76 different object parts.
Superclasses
Subclasses
Parts
bird cat cow
face, leg, neck, tail, torso, wings face, leg, neck, tail, torso face, leg, neek, tail, torso face, leg, neck, tail, torso face, leg, neck, tail, torso face, leg, neck, tail, torso
animals
dog
horse
sheep
bottle
body
indoor
potted plant
plant, pot
monitor
screen
person
person
face, arm & hand, leg, neck, torso
acroplane
body, engine, wheel, wings wheel
bicycle
hus
door, vehicle side, wheel, windows door, vehicle side, wheel, windows wheel
vehicles
car
motorbike
train
train coach, train head
Table 5: Part taxonomy of PASCAL-Part
D ADDITIONAL RESULTS AND ANALYSIS
Effect of Different Image Encoders Table 7a shows the comparison experiments of CLIP, MAE, and DINOv2, DINOv2 achieves the best performance on all datasets. Because the text-image contrastive pre-training limits learning complex pixel-level information, CLIP cannot precisely match image patches. Although MAE can extract pixel-level features by masked image modeling, it performs poorly. We suspect that the patch-level features extracted by MAE confuse the information about the surrounding patches, resulting in mistaken feature matching. In contrast, pre-trained by image-level and patch-level discriminative self-suuervised learning. DIVOv2 extracts all-purpose visual features and exhibit impressive patch-level feature matching ability. As a training-fres general perception framework, Matcher can deploy different image encoders. With the continuous development of vision foundation models, the capabilities of vision foundation models will continue to improve, and Matcher's performance and generalization ability will also be enhanced. This is confirmed by the continuous improvement in performance from MAE CLIP to DINOv2, demonstrating that Matcher has strong flexibility and scalability. Besides, we aim to make Matcher a valuable tool for assessing the performance of pre-trained foundation models on various downstream tasks.
14

---

Published as a conference paper at ICLR 2024
Fold
Parts
bench:arm, glass (drink container):base cellular telephone:bezel, guitar:body, bucket:body, can:body, soap:body. vase:body, crate:bottom, box:bottom, glass_(drink container):botton. basket:bottom, lamp:bulb, television television_set:buttonn set:button, watch:case, bottle:closure, book:cover, table:drawer, pillow:embroidery. car (atttoooiiilo).fedee dog:foot, bicyele:fork, bicycle:gear, clock:hand, bucket.handle, baskethandle, spoon:handle, bicycle:handlebar, guitar:headstock, sweater:hem, crashecn bucketrinner. body, hatrinner_side microwave bilcrovave_ovcniinnrr.neo Loven,innerssidee tray:inner_se side, pliers:jaw, shoe:lace, bench:leg, can:lid, fan:light, car (ar_(atttoobbleemiingg tray:outer_side, bicycle:pedal, can:pull_tabe shoe:quarter, can:rim, mugarim, pan_(for_cconl sir___oooiig)rimm tray:rim, basket:rim, car_ cuuttomobilerrunnnnghoon lamp:shade inner side, sweaterssoulder. television set:side, sweater:sleeve, blender:spout, jarsstcker, hel- met:strap, table:stretcher. blender:swicch, benchttable top. plastic bag:text, shoe:tongue, television_set.tpp. bicycle:top tube, hat:visor, car (automobile) car (pue
bowl:basc, handbag:base, basket:base, chair:base,
spoon:neck, sweater:neckhande
laptop_ compppttcmteeeeeeennn. chair:seat, bicycle:seat_st stay,
chair:apron, chair:back, bench:back, fan:base, cup:base, ____cmpuuerbbae__ppeeee s_steeepa_e_____e knife:blade, seissors:blade, bowl:body, sweater:body, handbag:body, mouse (computer_eq)) equipment:.body. towel:body. dog:body, bowlbottom, plate:bottom, televi- sion_ set:bottom, spoon:bowl, car (automobile) cellular lelephone:button, laptop_ computer:cable, fan:canopy, bottle:cap, clock:case, pipe:colied tube, sweater:cuff, microwave oven:dial, mug:drawing, scissors:handle, handbag:handle, handle
lap-
vase:foot, car_ (automobile): cupthandle, pan__for__apppppppaapppppaaaaaaa (ifffo/cooiigg////tooot/lee chandle, dog:head, bicycle:head tube, towel:hem, car_ (automobile :hhod, plastic_bag.innee bag:inner body, walletinner_ bbdyy glass_(drink, container)innee body, crate:inner side, pan_(for (ao_rfffoooooa wowp_/oooonninneerrrrrrr side, plate:inner_wwe _all,, soap:label, chair:leg, crate:lid, aapto__cmpputers hroomllowrrbrrstlee.. bristles, fan:motor, vase:eekk dog:nose, shoe:outsole, lamp:pipe, chairraill, bucketrrim, bowl.rim, car_ car_ (automobile))rim, bench:seat, bicycle:seat tube, soap:shoulder, box:side, carton:side, earphone:slider, bicycle:stem. chair:stile, benchestretcher, dog:tail, mug:text, bottletop, table:top. laptop computertouchpad. shoe:vamp. helmet.visor. car (automobile): mouse (computer equipment):wire
broom:lower
grre_ssticky_colttit ceclooo_oo__ppprrrrollll bicyclessaddle, seissors:serew,
2
table:apron, telephone:back _over, cover, plate:base, kettle:base, blender:base, bicycle:basket, fan:blade, plastic bag:body, trash casbbbdd. plate:body, mug:body, kettle:body, towel:border, mug:bottom, lelephone:button, microwave oven:control panel, microwave__ven::dor handle, dog:ear. hel- met:face shield, scissors:finger hole, wallet:flap. mirror:frame, kettle:handle, blenderchandle, tarphone-headlann, earphone:housing. bowl:inner_book trash can:inner body, helmet:inner side, basketrinner side, calculator.key, bottle:label, mouse coouse(ccoputer [computer_cquippeent]]) box:lid, trash_can:lid vase:mouth, pipe:nozzle, ladder:rail, hatrrim, platerrim, trash_ can:rim, bottlerring, (automobile) telephone:screen, mouse mouse_(compp (computer_equiie setttereeqnnttsssrooollll wheel, stool:seat, lamp:shade, bottle:shouldrr, wetttowwveeovenssiiiee basket:side, chair:spindle, hat:strap, belt:strap, car (automobilee)taillghh towel:terry_ _bar newspa- per:text, microwave_or hatpree__eeettimeeeeeeee dineeeddddly..aa shoe:toe_box _ox,, microwave_ovenne _oventtpp, car (autombbbiile):unn )ipper_(footwear)::: (footwear).vamp. car (automobile:
eeuuipenntyllltt_uutueeee button, dog:leg, dan:pedestal_colum
sweater:yoke
3
chair:rmm cemote_controlbbet television_setbasse, jar:base, tray:base, lamptbase, telephone:bezel, bottle:body, pen- bottle:bottom, tray:bottom, remote control:bution, bucket:cover, basket:cover, bicycle:down tube, earphone:ear pads, dog:eye, guitar:figerboard. blender:food cup, stool:footrest, scarf.fringes, knife:handle, vase:handle, car (aulomobile) teciinerrbodyy, cup:inner_bodyy body. box:inner_ side, carton:inner side. trash_ can:label, table:leg. bucket:loop. bottle:neck. dog:neck, stool:leg, jar:id, kettle:lid, car (automobiill))oggo pipe:nozzle_see stem, book:page, mouse (computer (computer_cquippmen) truip button, handbag.rim, jar:rim, glass_(drink baas__ddrin__cmmtttrrrritt container):rim, cup:rim, cellular cellular_telephoneeseeee telephome:screen, blender:seal_ lamp:shade, pappsscadd_ppppp.____e table:shelf, crate:side, pan_(for__cnlllpeeee cooking):side, mouse_ (computer_equuippppppee
cellular htaaulaareelephoncbbbk_..oo _ovoke_,,o,rr__oooooooooeeereeee bottle:base, bucket:base,
cil:body, scarf.body, calculator:body, jar:body, fiiss_ddrikkcoottannee pan aan_(forr__ wow__coooinxbbotoon
mug:inner body.
button, chairsskirt, car_ (uuoommollllyyyppsshhor.. hottle:spout, ladder:step, watch:strap, chair:stretcher, chair:swivel, can:text, jarttex, spoon:tip, slipper (footwear):tee box, blender:vapour cover, wheel, bieycle:wheel. car duutommbble):winddsieel handbag:zip
Table 6: Part taxonomy of PACO-Part
Effect of different types of prompts We validated the impact of different prompts datasets with scenes involving parts (PACO-Part), the whole (FSS-1000), and multiple instances (COCO-20*) in Table 7b: 1) level prompts needed for PACO-Part. which requires segmenting parts of an instance. However, our experiment results demonstrate that using instance-level prompts yields better results because instance-level prompts cover more situations than part-level prompts. 2) FSS-
15

---

Published as a conference paper at ICLR 2024
COCO-20" mean mloU
FSS-1000 mloU
DAVIS 2017
Encoder
Prompts
COCO-20'
PACO-Part
FSS-1000
J&F
Global
51.7
31.6
87.0
MAE 18.8
71.9
69.5
Part
51.7
30.2
79.2
CLIP
32.2
73,9
Instance
52.7
34.0
86.5
DINOv2
52.7
87.0
79.5
(b) Effect of different types of prompts.
(a) Effect of different image encoders.
Segmenter
COCO-20'
LVIS-92'
(SS-1000)
PACO-Part
SAM Semantic-SAM
52.7
31.4
87.0
32.7
51.1
30.1
87.5
36.0
(c) Effect different segmenters.
COCO-20'
LVTS-92'
FSS-1000
PACO-Part
Upper Bound
83.6
75.4
93.1
67.5
Matcher
52.7
31,4
87.0
32.7
(d) Upper bound analysis.
Table 7: Ablation study on the effects of different image encoders, different types of prompts, different segmenters, and upper bound of Matcher.
often involves one instance that occupies the entire image. Thus, global prompts are used for full image coverage. 3) For COCO-20', which requires detecting all instances in an image, instance-level points are the most effective. All the experiments are conducted on one fold in both three datasets.
Methods SegGPT
SAM DINOv2 #Params (M)
LVIS -2'
FSS 1000
307
17.5
85.6
base
base
180
28.6
85.3
large
base
399
29.9
85.7
Matcher
large
large
617
30.4
86.3
huge
large
945
31.4
87.0
(a) Ablation study model size.
Number FSS-1000
78.9
83.3
87.0
10
87.2
12
86.9
(b) Ablation study the cluster number.
Table 8: Ablation study on different model sizes and cluster number.
Ablation of model size Table 8a shows the results Matcher when using VFMs with different model sizes. When using SAM base and DINOv2 base, Matcher still performs well on various datasets and achieves better generalization performance LVIS-92' than SegGPT. Besides, the model size increases, Matcher can continuously improve performance.
Effect of different segmenters Table 7c shows the results when using Semantic-SAN (Li al., 2023) as the segmenter. Semantic-SAM achieves comparable performance with SAM on four benchmarks. Because Semantic-SAN can output more tne-grained masks, it performs better than SAM on PACO-Part. The results indicate that Matcher is general segmentation framework.
Upper bound analysis We conduct experiments on four different datasets and find that the upper bound of Matcher consistently outperforms the current performance on all datasets by a large margin Table 7d. This indicates that the Matcher framework has more potential. Therefore, Matcher can serve effective evaluation criterion for VFMs. assessing the performance of different vision models from a general segmentation perspective. Based on the advantage, Matcher can contribute to developing VFMs.
How does sho- segmentation work? the few-shot setting, we concatenate multiple references' features and match them with the target image in the PLM. The remaining process is the same the one-shot setting. Multiple samples provide richer visual details, enabling more accurate matching results and reducing outliers, resulting in performance improvement.
16

---

Published as a conference paper at ICLR 2024
Visualizations Fig. 6 shows the quality of background concept segmentation of Matcher. Fig. 7 visualizes the results of Patch-Level Matching, Robust Prompt Sampler and Instance-Level Matching. In addition, We provide more visualizations for one-shot semantic segmentation in Fig. 8, one-shot object part segmentation in Fig. and Fig. 10, controllable mask output in Fig. 1, and video object segmentation in Fig. 12. The remarkable results demonstrate that Matcher can effectively unleash the ability of foundation models to improve both the segmentation quality and open-set generality.
reference image
target images
Figure 6: Visualization of Matcher for the quality of background concept segmentation. Matcher can segment various background concepts like SegGPT.
reformnc
refoeree
PLM
ferwand maching
revens maiching
forward manding
reseme musding
RPS
point
poist prompta
ILM
12 Paom
TP makk
TP mask
result
saak
TP mask
TP mauds
Figure 7: Visualization of the results of Patch-Level Matching (PLM), Robust Prompt Sampler (RPS) and Instance- Matching (ILM). (a) For PLM, the Green stars present the correct matched points. and the Red stars present the matched outliers. The PL. can effectively remove most of the outliers via proposed bidirectional matching. (b) RPS can sample various point prompts by using the matched points of PLM. (c) Take the prompts inputs, SAM can output the mask proposals. Because there are still outliers in the matched points, SAM can output some false-positive (FP) masks. Thus, we propose ILM filter these FP masks and merge the true- mosiitii (TP) masks. Then, result. These components within the Matcher framework collaborate with foundation models and unleash their full potential in diverse segmentation tasks.
get the
17

---

Published as a conference paper at ICLR 2024
ER
Reference
GT
Prediction
Reference
GT
Prediction
Reference
GT
Prediction
Figure 8: Visualization of one-shot semantic segmentation.
18

---

Published as a conference paper at ICLR 2024
Reference
GT
Prediction
Reference
GT
Prediction
Reference
GT
Prediction
Figure 9: Visualization of one-shot object part segmentation PASCAL-Part.
19

---

Published as a conference paper at ICLR 2024
Reference
GT
Prediction
Reference
GT
Prediction
Reference
GT
Prediction
Figure 10: Visualization of one-shot object part segmentation on PACO-Part.
20

---

Published as a conference paper at ICLR 2024
Reference
GT
Controllable mask output
Figure 11: Visualization of controllable mask output.
21

---

Published as a conference paper at ICLR 2024
Figure 12: Visualization of video object segmentation.
22


Reviews:
[{"review_id": "Cqr3IsMoZF", "reviewer": "Reviewer_PEjB", "summary": "A perception paradigm is introduced name Matcher. It can segment anything by using in-context learning and two pretrained large vision models, which are DINO and SAM. There are three effective components within Macther: Correspondence Matrix Extraction (CME), Prompts Generation (PG), and Controllable Masks Generation (CMG). The whole pipeline is without training and showcases state-of-the-art  performance among other generalist models.", "questions": "Can Matcher segment a background concept just like SegGPT, such as blue sky or pavement? How does Matcher perform on more complex MOSE dataset for VOS?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The target task in this method is interesting to me. How be become a vision generalist is a hotspot in nowadays community. Matcher follows SegGPT to transfer off-the-shelf VFMs into a general segmentation model.\n\n2. This work constructs several new benchmarks for one-shot or few-shot in-context segmentation. They are challenging and meaningful to future works.\n\n3. The experiments demonstrate impressive generalization performance across various segmentation tasks.", "weaknesses": "1. The whole pipeline is too cumbersome to me. So many prompts seem unimportant from Table 7b in appendix. The final performance is significantly rely on the complicated ILM postprocessing from Table 4a.\n\n2. The impressive performance is likely due to an engineering ensemble of models. For example, the post merging of masks seems like a sort of output ensemble. Improvement from ensemble is common in previous segmentation works.\n\n3. I'm concerning about the inference time comparing to SegGPT, since Matcher needs to run the large-scale DINOv2 and SAM for every test image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNZsqBwUIg", "reviewer": "Reviewer_zVyp", "summary": "This paper proposes a general perception paradigm Matcher that utilizes off-the-shelf VFMs to address various segmentation tasks, such as few-shot/one-shot semantic/part segmentation. The VFMs in Matcher include DINOv2 as s feature encoder and SAM a promptable segmenter. Matcher involves no training, and achieves good results on several datasets.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Performance is good: Matcher achieves the highest scores over different segmentation benchmarks.\n* Every designed component is reasonable and works well from ablation studies.\n* Matcher doesn't need training. This improves its efficiency.", "weaknesses": "* This work seems like a combination of existing vision foundation models and a set of engineering tricks. The pipeline can be summarized as \"encode by DINOv2 -> select prompt (matching and sampling) -> prompt SAM -> select mask\". Although the result proves its effectiveness, the paper is more like a technical report and lack academic insight.\n\n* From the ablation in table 7(a), the performance of Matcher is largely influenced by different image encoders: MAE (18.8%), CLIP (32.2%), DINOv2 (52.7%). Does this mean the the selection of encoders would outweigh all the designs in Matcher?\n\n* More ablation is needed for the selection of promptable segmenters, such as SEEM and Semantic-SAM. As the author claims Matcher as a general paradigm, does the designs still work on different segmenters?\n\n* What would happen if the test image contains no reference concept? Can Matcher solve this situation, or output a false positive?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BYWQAv7IVg", "reviewer": "Reviewer_3ZxQ", "summary": "This paper introduces Matcher,  which utilizes pre-trained vision foundation models for various perception tasks without requiring specific fine-tuning or training. Matcher is capable of segmenting images using in-context examples, demonstrating its adaptability and proficiency across multiple segmentation tasks.", "questions": "1. Since SAM has already been trained on very large dataset, why not use SAM's encoder to generate propmt for itsself? If SAM has bad semantics, any intuision why is it?\n2. How exactly do you use DINOv2? Where layers' output do you use as the feature?\n3. How do you decide the number of clusters used do you use for Robust Prompt Sampler? I sould assume different objects may need different number of clusters. Any ablative study and explanation of these?\n4. It would be nice to show some visualizations of different examples how exactly the points prompts are filtered by Patch-Level Matching, Robust Prompt Sampler and instance level matching. \n5. To my understanding, the mistake made by Matcher depends on how good the semantics of the pretrained model to generate points and the mistake made by SAM itself. One interesting ablation study would be random sample same number of positive points on SAM as the number of points Macher generated, and take a look at the uper bound performance Matcher could achieve.\n6. How does few-shot(>1) segmentation work?\n7. For few shot segmentation, are the mistakes from localizations or bad mask shape? Some metric of localizaiton would be nice or visualizing more failure cases.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a good way to generate prompts for the visual fundation model Segment Anything Model (SAM). It is exquisite to plug-in and play with another pretrained vision model.\n2. The method do not require any training or fine-tunning.\n3. Extensive experiments and get reasonable results on many tasks.", "weaknesses": "1. Since there are a lot of engineering technique in the paper, one weakness is no open source code available. It would be nice to make the code public in the supplementary or in a public github repo. I would raise soundness score if the code is published.\n2. Some of the comparison in experiments section is not fair. For example, all the results from SegGPT[1] are used a smaller ViT-L backbone. Even though SegGPT's training data includes the Coco dataset, it's important to note that these pre-trained models like DINOv2 or CLIP have been pre-trained on much larger datasets. So it is hard to say if it is fair to compare such generalist model, but at least we should compare with model of the same size.\n3. Some of the details of implementation is missing. It lacks some of intresting abblation study. I will mention in quesions section below.\n\n[1] X, Wang et al. SegGPT: Segmenting Everything In Context. ICCV, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "018Cr0mj3V", "reviewer": "Reviewer_TD84", "summary": "This paper introduces Matcher, a framework that leverages pre-existing vision foundation models to tackle diverse perception tasks. The Matcher framework encompasses three key components: bidirectional matching for precise matrix correspondences extraction, a range of sample prompt designs encompassing part-level, instance-level, and global-level prompts, and controllable mask generation through instance-level matching. Through extensive experimentation on multiple benchmarks, such as COCO-$20^i$ and LVIS-$92^i$, the proposed method's efficacy is demonstrated. Furthermore, quantitative results highlight Matcher's ability to handle images in real-world scenarios, showcasing its open-world generality and flexibility.", "questions": "The running speed of the proposed method in terms of efficiency is an important aspect to consider. It would be valuable to compare the efficiency of the proposed method with related works to assess its performance in this regard. By conducting a comparative analysis, we can gain insights into how the proposed method fares in terms of running speed and efficiency when compared to existing approaches in the field.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and presents its ideas in a clear manner, making it easy for readers to follow the proposed framework.\n\n2. The idea of Matcher model is straightforward and practical. The three key components within the Matcher framework are not only effective but also highly efficient. (1) Bidirectional matching for precise matrix correspondences extraction; (2) A range of sample prompt designs encompassing part-level, instance-level, and global-level prompts; (3) Controllable mask generation through instance-level matching.\n\n3. The paper demonstrates good performance not only on standard one-shot benchmarks but also on a Video Object Segmentation (VOS) benchmarks.", "weaknesses": "1. One crucial aspect of achieving high performance in the Matcher framework heavily relies on the utilization of DINO-V2 for accurate correspondence matrix extraction. However, it is worth noting that this approach may involve assembling multiple foundation models, potentially leading to a trade-off between accuracy and efficiency.\n\n2. In light of recent research [1], it has been suggested that Stable Diffusion models offer a promising alternative for accurate correspondence matrix extraction, which has also been validated in the context of image-matching tasks. Consequently, it becomes pertinent to compare the performance of DINO-V2, the current method employed in the Matcher framework, with a diffusion-based correspondence extraction approach. Such a comparative analysis would enable a comprehensive evaluation of the two methods, shedding light on their respective capabilities within the Matcher framework and determining which approach yields superior results.\n\n[1] Tang et al. Emergent Correspondence from Image Diffusion, in NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)