Multi-instance Interactive Segmentation with Self-Supervised Transformer
Authors: Xavier Jiménez
Jaonary Rabarisoa
Valentin Belissen
Quoc Cuong PHAM
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision Transformer
Self-supervised learning
Interactive Image Segmentation
Semi-supervised learning

Abstract: The rise of Vision Transformers (ViT) combined with better self-supervised learning pre-tasks has taken representation learning to the next level, beating supervised results on ImageNet. In particular, self-attention mechanism of ViT allows to easily visualize semantic information learned by the network. Following revealing of attention maps of DINO, many tried to leverage its representations for unsupervised segmentation. Despite very promising results for basic images with a single clear object in a simple background, representation of ViT are not able to segment images, with several classes and object instance, in an unsupervised fashion yet. In this paper, we propose SALT: Semi-supervised Segmentation with Self-supervised Attention Layers in Transformers, an interactive algorithm for multi-class/multi-instance segmentation. We follow previous works path and take it a step further by discriminating between different objects, using sparse human help to select said objects. We show that remarkable results are achieved with very sparse labels. Different pre-tasks are compared, and we show that self-supervised ones are more robust for panoptic segmentation, and overall achieve very similar performance. Evaluation is carried out on Pascal VOC 2007 and COCO-panoptic. Performance is evaluated for extreme conditions such as very noisy, and sparse interactions going to as little as one interaction per class.

Under review as a conference paper at ICLR 2023
MULTI-INSTANCE INTERACTIVE WITH SELF-SUPERVISER TRANSFORMER
SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
The rise of Vision Transformers (ViT) combined with better supe-suue learn- ing pre-tasks has taken representation learning to the next level, beating supervised results on ImageNet. In particular, self-attention mechanism of VIT allows eas- ily visualize semantic information learned by the network. Following revealing of attention maps of DINO, many tried to leverage its representations for unsuper- vised segmentation. Despite very promising results for basic images with single clear object a simple background, representation of ViT are not able to seg- ment images, with several classes and object instance, in an unsupervised fashion yet. In this paper, we propose SALT: Semi-supervised Segmentation with Self- supervised Attention Layers in Transformers, glass/multi-instane segmentation. We follow previous works path and take a step further by discriminating between different objects, using sparse human help to select said objects. We show that remarkable results are achieved with very sparse labels. Different pre-tasks are compared, and show that supe-supe ones are more robust for panoptic segmentation, and overall achieve very similar performance. Evaluation is carried out Pascal VOC 2007 and COCO-panopte Performance is evaluated for extreme conditions such as very noisy, and sparse interactions going little as one interaction per class.
interactive algorithm for multi-
INTRODUCTION
The last ten years have seen the rise of computer vision tasks such as localization and segmentation. As a result, technologies such as autonomous driving or robotics have met great success at the expense of annotating huge enough datasets. Indeed, state-of-the-ar approaches are all based on training neural network in supervised fashion (Strudel et al. 2021 Xie et al. (021). Although this might work well areas where there are enough resources to label million images, there others where there are almost labels but data is already available in large quantities. For instance, in fields such as astronomy, sometimes one is limited by the amount available ground truth labels (Pasquet et al., 2019). On other ones like medical imaging, data needs to be labeled by professionals, which is very expensive. Therefore, leveraging unlabeled data is a necessity in many computer vision tasks.
Numerous attempts exist in the literature to solve this problem, such as semi-supervised learning (iipf & Welling 2017), weakly supervised learning (Strudel et al. 2022). and active learning (Agh- dam et al. 2019). These methods can achieve some improvement, but still need slight supervision. More recently, self-supervised pre-tasks have leveraged representation power of Vision Transformer (ViT: Dosovitskiy et al. 1021) in a similar fashion to what has been done in NLP. Indeed, an image can be seen as a sequence p x p patches. Transformers have recently outperformed convolutional neural networks, and results with self supervised pre-task DINO (Caron et al.] (021) have shown impressive salient regions in the attention maps from the class token in the last ViT layer. This has led authors to test the unsupervised foreground detection capabilities of such representations (Wang) jt al. 2022. Amir et al. 202I). Authors tried clever ways to cluster these feature representations to split foreground and background regions. Melas- et al. (2022) went step ahead and tried to do this for more than one foreground object. However, these applications are limited to simple images with clear background and very few salient objects.

---

Under review as a conference paper at ICLR 2023
We believe that the representation power of self-supervised ViTs can be pushed much further with very sparse human interactions. Here, we try to discriminate between different objects, using sparse human help to select said objects. Our goal is twofold, want assess the representation power of self-supervised ViT', and at the same time create an interactive segmentation algorithm that is not powered by supervised learning algorithm, but still harnesses information from dataset. Indeed, if one has huge unlabeled dataset, then self-supervised learning could be first used to derive meaningful representations, and our algorithm could help label images in just few seconds.
In this paper, we propose SALT: Semi-supervised Segmentation with Self-supervised Attention Lay- ers in Transformers. A graph based sumi-supervised approach that harnesses the representation power of self-supervised ViTs to create segmentation masks from sparse human interactions. We test segmentation performance on Pascal VOC 2007 (Everingham et al., 2010p, and modified ver- sion of COCO COCO-panoptic (Lin et al. 20155) that contains only big things, hereafter COCO big things. However, because of the nature of interactive segmentation algorithms themselves, we will not be comparing to other algorithms. Indeed, the major obstacle is that algorithms usually take different forms of inputs, and can also be iterative. To the best of our knowledge, there are no interactive seg- mentation methods that share the same input as ours. For each dataset, will create interaction dataset with human inputs, and we will craft elaborated evaluation protocols that show what can be realistically expected from our algorithm, as well as its limitations.
We study the performance of our algorithm in extreme conditions, which can be interesting for some unsupervised tasks where we have very sparse information of the position of object. To the best of our knowledge, this is the first unsupervised interactive segmentation algorithm that is able to handle many panoptic classes simultaneously, while achieving pleasing results. Although results are still far below the performance of supervised state-of-the-art algorithms, this work shows the potential of future ViTs for zero-shot interactive segmentation, and eventually unsupervised segmentation.
The paper is organized as follows. In section 2 we present the prerequisites. In section 3 we explain our method. In section 4 we present the modifications made COCO-panoptic, and how we gathered interactions. In section [5] we compare different pre-trained ViT's, and evaluate the robustness of our method to noise and interactions sparsity up to one patch per class. Finally, in section 6 we present our conclusions,
2 RELATED WORK
Vision Transformer. Transformer architecture (Vaswani al. 2017) has become the default archi- tecture for natural language processing (NLP) since it was first introduced five years ago. It was not until very recently that computer vision started transitioning from Convolutional Neural Networks (LeCun et al. 1998) to Transformers. Pioneer works tried to implement the self-attention mechanism within CNNs (Hu et al. 2019 Ramachandran et al. 2019 Zhao et 2020). Dosovitskiy et al. (202T) ultimately released the Vision Transformer, using 16 x 16 patches as tokens, and almost the same coder as in the original Trasformer. Since they first appeared, a lot of strategies have been developed to train ViT's more effectively (Beyer al. 2022 Touvron et al. 2021 2022). as well as variants (Liu) let al. 2021). Besides, many recent work have shown that ViTs trained in a self-supervised fashion (Caron al. 2021 Bao et al. 2021 Assran al. 2022) outperform their supervised counterpart. Self-supervised ViT attention maps have also shown high semantic comprehension.
Self-supervised learning. In recent years, different clever pre-task have been developed exploit unlabeled data and pre-train models in a self-supervised fashion. Pioneering works designed inge- nious pretext tasks exploit internal structures of data, such as patch ordering prediction (Noroozi) & Favaro 2017), recovering colors from grayscale images (Zhang et al. 2016), image rotation pre- diction (Gidaris al. 2018), etc. Nowadays, most approaches fall into one of two categoriers: generative or discriminative. Generative methods are usually based on masked image encoding (He) ct al. 2021 Bao et al. (02I). Contrastive learning usually uses siamese networks to discriminate between two views of image (Chen et al. 2020 He et al. 2020 Grill et al. 2020 Caron al. 2021
Assran et al. 2022).
Unsupervised Segmentation. Earlier methods mostly used color/background constraints (Cheng ct al. 2014 Wei et al. 2012). Recently, methods based on extracting features using a self-supervised Transformer (Dosovitskiy et al. 2021) based DINO (Caron et al., 2021) significantly improved
2

---

Under review as a conference paper at ICLR 2023
g
Figure 1: Sketch of how the graph is built, and how the human interactions are used. Here, each node corresponds ViT patch, and each color different class. Then, the graph with sparse labels goes through label propagation (IPP) that predicts unlabeled nodes.
over state-of-the-arr for unsupervised object discovery and segmentation (LOST: Siméoni et al. 2021 TokenCut: [Wang et (022). These methods use ViT features create graph G(V, E) where nodes are patches, and edges are defined according to the similarity of the features. However, these methods cannot work images that have more than one salient object. More recently, MMela- Kyriazi et al. (2022) generalized this approach to more than one object. However, there is much work to do before these methods can properly segment a complex multi-instance image.
Interactive Segmentation. Interactive segmentation methods attempt to use external information to extract foreground objects in a complex environment whose background cannot trivially sub- tracted. This information
be passed in many different ways, mainly (i) extreme points (ii) bounding boxes, (iii) scribbles, (iv) positive/negative points. Earlier methods used color statistics (Magic Wand: Gutiérrez & Vexo Bayes matting: Yung--u Chuang al. 2001). edge contrast (Intelligent Scissors: Mortensen & Barrett 1995). Maxflow optimization (GrabCut: Rother et al. 2004 Saliency Cuts: Fu al. 2008 Bai Wu 2014), optimal transport (Rabin & Papadakis 2015))
These methods are all limited by the information contained on the image, and as a result, they often need many interactions to achieve satisfactory result. Deep learning based methods use point-based interactions such as extreme points (Xu et al. 2016 Papadopoulos et al. [017 Zhang et al. 2220),
positive/negative points (Mahadevan et al. 2018 Hao et al. 2021 Zhang et al. 2022). and as little as two points (UCP-Net: DDupont et al. 2021). However, there still does not exist multi-instance interactive segmentation algorithm that leverages information contained in a dataset.
APPROACH
The general pipeline of our approach is summarized in Figure [ A user draws scribbles to select different objects in an image. Then, the patch feature representation are extracted from the pretrained VIT and a graph G built. Finally, the user labels are used along with as input for the Label Propagation (LP: Zhou et 2004) algorithm, which predicts the remaining labels.
Label propagation. Unlike unsupervised methods, we want to explore the limits of self-supervised representations in more complex scenarios. By using sparse human interactions, we can therefore use more complex images with lots of objects/instances. As a result, we are a semi-supervised learning scenario. Label propagation (Zhu & Ghahramani 2002) Zhou et al. (004)) are a kind of graph based semi-supervised learning algorithms. The underlying idea is that since we do not have labels for each node, known ones ares used to propagate information through the graph and label remaining nodes. Let X = (I1, learning, only have fraction X, (#1...... 21), for which labels Yi = (1.)) provided, and fraction X,, (#1+1,.... z1+w) for which we do not know the labels. The algorithm computes the normalized graph Laplacian matrix and predicts labels Y (Yi, Y,,) iteratively. The algorithm pseudo-code is presented in Algorithm [
In)) be our dataset with n examples. In semi-supervised
The cost function associated with this problem is
(I) where A the affinity or adjacency matrix, D the diagonal degree matrix, and μ > 0 is the regularization parameter. The first term is the smoothness assumption, which ensures consistency
c(t) = 2
Σ Asj
0 VD
+ 14
10: - 2ill2
DDJ
3

---

Under review as a conference paper at ICLR 2023
Algorithm Label spreading (Zhou et al. 2004)
Require: a E [0, 1[
Compute affinity matrix A, Aii +0
Compute diagonal degree matrix D: Dú: f- E, As As
Compute normalized graph Laplacian + D D-1/2 AD-1/2
Initialize ( (yl..... ................................ Md for /. in range Nater do (++1) + aCy(t) (1) + (1 a)
if convergence then break end if end for return Label x3 as a label argmax D (f00)
with the geometry of the data. The second term is the fitting constraint, which penalizes rapid changes in between points that are close. Zhu & Ghahramani (2002) forces y; = Y,. However, if there is noise on the labels, the algorithm should be able to re-label them. The same kind of cost criterion is obtained for spectral clustering for the relaxed NP-hard problem of minimizing the normalized cut of g.
Feature representation. Our goal is to leverage self-supervised learning methods representation power. It has been shown that Vision Transformers pretrained in such ways create powerful rep- resentations that can be used to segment salient objects in images (e.g. [Siméoni et al. 2021] Amir eet al. 2021] Wang et al. 2022 KyriaziK et al. 2022). These methods are all based on some clustering algorithm that will separate the salient object from the background. All of these methods are based on DINO (Caron et al. 202T), and conclude through an ablation study that the best feature representation the keys from the last ViT-B layer. Here. will use Masked Siamese Networks (MSN: Assran et al. 2022), a very similar pre-tasks to DINO. We will also use keys from the last VTT-B/16 layer, although we found significant improvement compared other representations in the last layers (see Appendix(A)
Graph construction. Following Siméoni al. 202 use VIT features build graph GVV E) where each node V represents a patch. We consider three different approaches. (i) We define edges Eig between nodes V. V, as (Wang al. 2022)
Ei - {
if S(i,)) 2T otherwise
(2)
where S is similarity score based the cosine similarity of the feature vectors of the two patches, T ejo, 1[ is a Gaussian kernel such that the adjacency matrix is
and is very small to ensure the graph is fully connected. (ii) A
|r: x, 11
Aij eap
20²
(iii) A K-nearest neighbors graph. We will refer to these graphs as similarity, RBF, and KNN re- spectively.
We also tried to add low level features such color information to the graph, as in Melas Melas-Kyriazi ct al. (2022)
A A feat AA
where A feut is the adjacency matrix for the graph defined using ViT features, À is a hyperparameter, and Acotor is a KNN graph defined using color information. However, did not show any improve- ment in results. Experiments will be carried out using RBF graph with color information, and an ablation study is available in Appendix B
Higher resolution. We propose effectively double the patch resolution by computing the lin- ear interpolation between patch representations. Amii et al. (202I) proposed to take overlapping patches. Indeed, by passing the image through single convolution layer, with stride that equals
4

---

Under review as a conference paper at ICLR 2023
images from the validation set. (middle) COCO-panopios labels. (right) COCO- CocO-panoptic Big Things labels: stuff and small things are merged with the back- ground.
Figure 2: (left) Two
half the patch size, we achieve overlapping patches, and patch resolution half the original size. However, this comes at the expense of GPU memory, and only achieves slightly better results (mloU 81.3 instead 81.0 on MSN for VOC07). ViT-B/16 with resolution increase. However, this usually achieved worst results. For instance, achieved mloU 76.8 on DINO-B/8, and 80.8 on DINO-B/16 for VOC07. Therefore, we did not these techniques and used our linear interpolation instead.
also possible to use ViT-B/8 architecture instead of
Even if this allows us to reduce the patch size by half, we still need a way to achieve pixel-level segmentation. We cannot decrease to resolution below p 8 as it would become computationally too expensive. Krähenbühl & Koltun (0122) developed a method to label every pixel in the image with one of several predetermined object categories. Then the problem is solved a posteriori (MAP) inference in a conditional random field (CRF) defined over pixels or image patches. Here, our patch-level mask as input and mask 20% patches with uniform random distribution.
maximum
4 EVALUATION
4.1 DATASETS
We will carry out evaluations on Pascal VOC 2007 (Everingham et al. 2010) train-val set, and COCO-panoptic (Lin al. 2015a) validation dataset. COCO usually has many complex scenes with multiple objects. There are two main categories of objects: things (e.g. persons, indoor objects, etc.) and stuff' (e.g. grass, sky, etc.). Manually labeling all these objects would be very time expensive, and would bring little additional information regarding the performance of our model. Therefore, we keep only things. Besides, we remove objects with less than 0.5% of total pixels, small objects are also time expensive to label. We remove images for which there are no remaining objects. These modifications are performed on the validation set, and we call our subset COCO- C-CC-ppoppiicc big things. Figure 2 shows an example of the COCO-panopie dataset before, and after our modifications.
4.2 INTERACTIONS
Properly evaluating an interactive segmentation algorithm is very subtle task. Depending on the kind of interaction, they can simulated fairly easy. For instance, simulating extreme points (Pa- padopoulos et al., 2017), or sparse point interactions (Dupont et al.) (021) can be done with very little assumptions. However, simulating more complex interactions such as scribbles an active field of research (Jiag et al.) 2016). Besides, the amount of interactions is not the only important parameter. Indeed, one should also consider the time spent to label an image, for not all interactions take the same time. For instance, a straight line can be done faster than aiming for particular point. However, time cannot be easily compared between annotators, and we believe using it metric would not bring any relevant information. Therefore, here we mostly focus on the interactions alone.
5

---

Under review as a conference paper at ICLR 2023
Figure 3: COCO- Panopptc results. From left to right, image, ground truth, handmade interactions, predicted labels (patch scale), and predicted labels (pixel scale).
To the best of our knowledge, there no effective method that properly simulates human inter- actions. Therefore, we decided to manually annotate these datasets, for there no better way to evaluate human interactions than actually asking humans to do them. We tried to give simple in- structions to annotators, so that annotations remained truthful. We developed a custom software to annotate images according to their real labels. We asked annotators to spend around fifteen seconds per image on COCO, and less than five on VOCO7. The instructions read follows
Do not spend more than few seconds per image.
If the scene is complex, try to draw scribbles in regions of interest that contrast with objects.
You do not have to label all the different components of instance (for semantic labels only), but you can if you want as long as it does not take too much time.
You can draw scribbles but you can also click once if you feel the area is too small.
You can go the edges if the area is too small, but try to keep within the area as much as possible.
Interactions for VOCO7 were made by one person only (4 seconds per image on average). Interac- tions for COCO-panople where made by 15 different people (16 seconds per image average), and amount to 1423 out of 4823 images.
EXPERIMENTS
this section we present different experiments we performed to evaluate our method. First, we compare different pre-trainings and see how they perform. Then, study the robustness of our method against noise, as well as against using a fraction of interactions. Finally, we test our algo- rithm under extreme conditions using only one patch per class. We use the t timm (Wightman) 2019) library for ViT architecture, and Scikit -eriitteerrn implementation of the Label spreading algorithm (Zhou al. 2004

---

Under review as a conference paper at ICLR 2023
5.1
IMPACT OF THE PRE-TRAINING
82.0
81.0
80.5
80.0
79.5
79.0
VIT-0/16
78.5
VI
WF-33
DINO
MAE
4G8
SE0
SECO-PACCAL
McDo
Figure 4: Pascal VOC 2007 analysis over all pre-
State-of-the-ard methods in unsupervised segmentation (Siméoni et al. 2021 Amir et al. 2021) showed through ablation studies that using keys from the last ViT-B/I6 layer patch represen- tations yielded best results for DINO pre-trainings. We conducted an ablation study of our own in Appendix A and found that the deeper layer, the better result. However, the gap between keys and queries or values is very small and does not suggest that keys are better overall. Still, we will conduct analysis using keys from the last VIT layer, for both base and large architecture.
We compare results for different kind of pre-trainings on ViT-B/16 and ViT-L/16. We use vanilla ViT pretrained on ImageNet- Ik and ImageNet-21k (Dosovitskiy al., 202T): CLIP (Radford al. 2.21 text-to-image pre- pre-training that jointly trains an image encoder, and text encoder to predict the correct image/text pairs; Self supervised methods pre-trained on ImageNet- 1k: DINO (Caron al. 2021). MAE (He et al. 2202), MoCo-v3 (He et al. 2020), and MSN (Assran et al., 2022); and ViT encoders trained with Segmenter (Strudel et al. 2021), a supervised method for image segmentation, ADE20k and Pascal Context.
Figure shows results for Pascal VOC 2007. We can see that there is a clear gap in performance between vanilla ViTs and self-supervised methods. Also, Segmenter ViT beats self-supervised meth- ods for this task. Figure shows results for ConO-panoptic using panoptic, and semantic label We that the gap between vanilla pre-trainings and self-supervised ones increases for panop- tic labels. Moreover, Segmenter ViTs are on par with self-supervised ones for semantic labels, but beaten for panoptic ones. Therefore, self-supervised pre-trainings seem be better suited panoptic segmentation.
5.2 NOISE ROBUSTNESS
An important feature of our algorithm is its capacity to perform well even if there are incorrectly labeled patches within the interaction. Our handmade interactions already contain wrongly labeled patches, as seen in Appendix C Our goal here is to analyse how much we can deteriorate the interactions, while still achieving good results. To so, we add noise to each handmade interaction independently. The background is usually labeled without problems. Therefore, we do not add noise to it.
For each mask in a label, we define density map P for each patch
ds(x)²
P(x) = exp
if S
20
(4)
0
otherwise
'Semantic labels are created from panoptic ones. We do not the existing COCO labels for semantic segmentation, which slightly different.

---

Under review as a conference paper at ICLR 2023
Panoptic
Semantic
80
80
mloU
miou
74
74
VIT-0V16
VIT-BV2.66 VITULGG
V17
GLP
DIN
MAB
MSW
sEG-ADE20K
SEG-PASSAL
viT
C.P
DWN
MAE
Meto x3
HSN
SES-ADEzOK
SEG-PASCAL
Hete
Figure 5: COCO- CeCO-ppaoptic v. semantic analysis over all pre-trainings.
where S is the mask, is parameter set to 4 to keep the simulation realistic, and ds(x) is the shortest distance between æ and S. We sample noisy patches from this distribution.
Results for COCO-panoptes are shown in Figure 6 For each interaction, percentage p E [0, 0.5 of unlabelled patches are added to the interaction uniformly at random. This percentage corresponds fraction of the number of labeled patches per object. For instance if noise 0.1, then for each interaction, unlaheled patches accounting for 10% the total number of labeled patches are added the interaction. Overall, the algorithm does not seem to be robust extra noise. However, we usually do not have more than 5% extra noise per category. Thus, our algorithm is robust enough for interactive segmentation.
5.3 SPARSE LABELS ROBUSTNESS
80
Semantic
80
Panopiiic
75
mloU
60
Semantic ntic
Panoptic
0.1
0.2
0.0
0.6
2.0
Fraction of added noise
Fraction of removed Interactions
Figure 6: (right) COCO noise robustness analysis. (left) Analysis of robustness against using fraction of the handmade labels. The masked % corresponds the fraction of labeled patches that is removed in an image for each individual instance.
We test our model robustness against using a fraction of the handmade labels. For each instance, we remove uniformly at random p% patches from the interactions, always leaving at least one patch instance. Figure shows results for p [0, 0.99]. The algorithm does not suffer a drop in performance until we remove more than 60% patches per instance, and performs remarkably well until 80%. The drop is more important for panoptic labels than for semantic ones. Indeed, using more labels forces the algorithm to properly separate two instances of a same class. If one leaves only few labels per instance, the algorithm could be inclined discriminate between the different
8

---

Under review as a conference paper at ICLR 2023
parts of the objects instead (e.g. separating the head from the body). Hence, the algorithm needs more labels to achieve good results in panoptic configuration.
5.4 ONE PATCH RULE THEM ALL
We simulate interactions uniformly at random using one labeled patch per class, including the back- ground. This is the lowest amount of interactions that we can give to our algorithm, and as a result, the most extreme conditions. Using only one patch per class means that if we have an algorithm that finds seeds for different objects, this is the highest performance we could expect for said unsuper- vised algorithm. Figure 7 shows results for few images. We can see that using only one patch in an image gives many possible correct labels. For instance, on the second image from Figure 7 there many objects but only three are labeled. However, the algorithm selects them all and spreads them accross the three labels. Therefore, the algorithm would need accurate labels for all objects in image be properly evaluated. We performed this experiment five times and achieved mlou of 61.7 0.5 for VOC07, and 50.3 + 0.8, 56.7 + 0.7 respectively for COCO panoptic and semantic. Thus, the algorithm performs remarkably well in extreme conditions.
Figure 7: Same as Figure with one patch per class.
CONCLUSION
In this paper, we have introduced a method, SALT, for multi-instance interactive segmentation lever- aging self-supervised learning, and Label Propagation. This work goes along the path of previous works that show the power of graph theory. We have shown that self-supervised representations can accurately distinguish, with relatively sparse human interactions, between different classes and instances within a class. We achieved this by evaluating performance on two datasets: VOCO7 and a modified version of COCO panoptic that only includes big things. Our main results in this work include (i) Self-supervised learning pre-trainings are more robust for panoptic segmentation (ii) the algorithm is not very robust to noise, but is good enough for the expected amounts of noise (iii) the algorithm can achieve very good performance with very limited interactions (iv) the algorithm performs remarkably well in the most extreme conditions using only one patch per label. All in all, with better pre-tasks come, this technique will help label datasets much faster, and this method could also generalized for unsupervised segmentation if coupled with an object detection algorithm.
believe that

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hamed H. Aghdam, Abel Gonzalez-Garcia, Joost van de Weijer, and Antonio M. López. Active Learning for Deep Detection Neural Networks, November 2019. URLattt://acxiv.rrr arXiv: 1911: 1911199111.968 [cs].
abs/ 1911. 09168
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT Features as Dense Visual De- scriptors. arXiv:2112.058// [cs], December 2021. URL http://arxiv.orrg http/abs/2111
05814 arxiv: 2112.05814.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vin- cent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label- Efficient Learning. arXiv:2204.077 071104 [cs, eess), April 2022. http://arxiv.
abs/ / 2204 07141
arXiv: 2204.07141.
Junjie Bai and Xiaodong Wu. Error-Tolerat Scribbles Based Interactive Image Segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 392-399, June 2014. doi: 10.1 00.100//VVP..0044.57 ISSN: 1063-6919.
Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT Pre-ninnning of Image Transformers.
arXiv:2106. 08254 2106.08254.
00650824 [cs], June 2021. URL ottp://aarx...orrgaab////2..... arXiv:
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain ViT baselines for ImageNet- lk. Technical Report (ixXxx:2005001880 arXiv, May 2022. URL 2205. 01580 arXiv:2205.00 580
abs/
[es] type: article.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self- Superviue Vision Transformers. arXiv:2104.1 14294 [cs], May 2021. URL http:::t:///i...oooo http://////aaa/////////////
14294
arXiv: 2104. 14294.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning Visual Representations. arXiv:2002.05570 0,5009 [cs, stat), June 2020. URL
http://arxiv v. orgg abs /0los/2002 2002 05709
arXiv: 2002.05709.
Ming-Ming Cheng, Niloy Mitra, Xiaolei Huang, Philip H H Torr, and Shi-Min Hu. Global Con- trast based Salient Region Detection. JEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, pp. 14, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010. 11929 [cs], June 2021. URL arXiv: 2010.11929.
abs 11929
Camille Dupont, Yanis Ouakrim, and Quoc Cuong Pham. UCP-Net: Unstructured Contour Points for Instance Segmentation. arXiv:2109.077 109907792 [cs], September 2021. URL http: org/ abs//bb/////// 2109. 07592
arXiv: 2109.07592.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. Int J Comput Vis. 88(2):0033338, June 2010. ISSN 0920-569, 1573-1405. doi: 10. sprinnge wtw. 10. 1tt.//1..1101/0000
URL http::/iink..
1007 -22-
Yu Fu, Jian Cheng, Zhenglong Li, and Hanqing Lu. Saliency Cuts: An automatic approach object segmentation. In 2008 19th International Conference on Pattern Recognition, pp. 1-4, December 2008. doi: 10. 0009/CCPR.208.47613355
ISSN: 1051-4651.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised Representation Learning by Predicting Image Rotations, March 2018. URL http: /ttt:///rxiv, arxiv. [cs].
0788033.77720 07728
10

---

Under review as a conference paper at ICLR 2023
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised Learning, September 2020. URL http:
w/wpxiv.orgg/ab//////// 07733 arXiv:2006.07733 [cs, stat].
Mario Gutiérrez and Frederic Vexo. The Magic Wand. 2003. doi: 10.114559849552.849972
Yuying Hao, Yi Liu, Zewu Wu, Lin Han, Yizhou Chen, Guowei Chen, Lutao Chu, Shiyu Tang, Zhiliang Yu, Zeyu Chen, and Baohua Lai. EdgeFlow: Achieving Practical Interactive Segmenta- tion with Edge-Guided Flow, October 2021. Number: arXiv:2109 09.499999964.. arXiv:2109.09400 [cs].
org http://bs///./.... 09406
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshiek. Momentum Contrast for Un- supervised Visual Representation Learning, March 2020. URL http://azxiv. www.voorgaaaae abs/ 1911. 05722 arXiv:1911.05722 [cs].
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners December 2021. URL hattp://arxiv.or
abs/2111.0633 06377 arXiv:2111.06377 [cs].
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local Relation Networks for Image Recog- nition, April 2019. URL /tLtttp://aaatt
http//aaxiv..v wttp://wwwwwg////////////. 114991
11491 arXiv:1904, 11491 [cs].
Bingjie Jiang, Tongwei Ren, and Jia Bei. Automatic Scribble Simulation for Interactive Image Segmentation Evaluation. pp. 12, 2016.
Thomas N. Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Net- works, February 2017. URL htttp://axxx...oorr/a////O00 02907 arXiv:1609.0290
[cs, stat].
Philipp Krähenbühh and Vladlen Koltun. Efficient Inference in Fully Connected CRFs with Gaus- sian Edge Potentials, 12X.v:1210.560 [cs], October 2012. URL 1210.5644 arXiv: 1210.5644.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Ha. Gradient-Based Learning Applied to Document Recognition. pp. 46, 1998.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pictro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. ar05:: 12 [es], February 2015a. URL W.RL.tttpp//axxi..rr//// 1405. 0312
arXiv: 1405.0312.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Com- mon Objects in Context, February 2015b. URL http:/ pttt:://axxiv.oom http///awvvv....g////// wwwpvv..rrg/bb//445..5 0312 [es].
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. 20t.:2/03./4.33 [cs], August 2021. httpttpp://aa....oro/bbb.bs///
14030 arXiv: 2103.14030.
Sabarinath Mahadevan, Paul Voigtlaender, and Bastian Leibe. Iteratively Trained Interactive Seg- mentation. 1805 04398 [cs], May 2018. URL http:://rxxiti wrw. wwwp/br/////aoooeo httpaaass/1150 04398
arXiv: 1805.00 8805.04398.
Luke Melas -arrazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization. Tech- nical Report Nw0.v::22550073330 arXiv, May 2022. URL http::// httpvxiiiivv//////v//ea/////aavv abspg//5b...... 07839 arXiv:2205.07839 [cs] type: article.
11

---

Under review as a conference paper at ICLR 2023
Eric N. Mortensen and William A. Barrett. Intelligent scissors for image composition. In Pro- ceedings of the 22nd anmual conference on Computer graphics and interactive techniques SIGGRAPH '95, pp. 191-198, Not Known, 1995. ACM Press. ISBN 978-0-89791-701-8 doi: 10.1 0..1145/21830.218888 URL httph:t/://pottal wtt.:aaw.w...cctt.aoooooom co
218380 218442
Mehdi Noroozi and Paolo Favaro. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles, August 2017. URL attp::/arxix: wtwp://....///aa/1....00300
arXiv:1603.09246 [es].
Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vittorio Ferrari. Extreme clicking for efficient object annotation. arXiv: 0778002750 [cs], August 2017. URL http://arxiv.org
abs/1708 02750
arXiv: 1708.02750.
Johanna Pasquet, E. Bertin, M. Treyer, S. Amouts, and D. Fouchez. Photometric redshifts from SDSS images using a convolutional neural network. A&A, 621:A26, January 2019. ISSN 0004- 6361, 1432-0746.
doi: 10. 10.1051100 100.006/000466666/20133331111111 18.1052.0 0004 10.1061/000--3331220103333
URL https: //wp://wwwwwww//. aanda org/
Julien Rabin and Nicolas Papadakis. Convex Color Image Segmentation with Optimal Transport Distances. arXiv: met::00003000 [cs], March 2015. URL http://aarx.v..rrg//b//SS000
01986 arXiv: 1503.01986.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.0002 [cs]. February 2021. URL httt:://aax/et httE:://arxlv.orge www.arwwww.rgga////a/aaaaaa wttp://ba////.../////0/ arXiv: 2103.00020.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-Alone Self-Attention in Vision Models. Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URLhttts://papeeee nips httppsccccpppoe cc/paper/
2011/hhahh/311a7554ccaaa
f2afc Abstract Abstaattttttttrttttttt. htmm
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "GrabCut" - Interactive Foreground Extraction using Iterated Graph Cuts. pp. 6, 2004.
Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing Objects with Self-Supervised Transformers and no Labels. arXiv:2 109.0099..... /4279 (cs), September 2021. URL http:://rrxiv.org//eee 2109. 14279
arXiv: 2109.14279.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for Se- mantic Segmentation. arXiv:2 (at::2/05.0000 105.05633 [cs], September 2021. URL http:://axxvv..rrg arXiv: 2105.05633.
abs/2105. 05633
Robin Strudel, Ivan Laptev, and Cordelia Schmid. Weakly-supervised segmentation of referring
expressions. arXiv:2205.0477 [cs], May 2022. URL http:///axiiii wwtp://wwwwgg/////////aaa
04725 arXiv:
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers & distillation through attention.
1rXiv:2012.12877 [cs], January 2021. URL http://wxx...rr/ab///222111111 arXiv: 2012.12877
Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge of the VIT. arXiv:2204.071118 [cs], April 2022. URL /uttt::/aarae httt:::/arxxio.or/abbb////// wwwp:i/wwwrr.bb///eeae.eoo /204 07118 arXiv: 2204.07118.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.023 [cs], De- cember 2017. URL wtw.:ttppp0 hhtt:://aaxiin http///t/////artt.oorr///s////// 1706. 03762 arXiv: 1706.03762.
12

---

Under review as a conference paper at ICLR 2023
Yangtao Wang, Xi Shen, Shell Hu, Yuan Yuan, James Crowley, and Dominique Vaufrey- daz. Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut. 1rXiv:2202.11539 [cs, stat), March 2022. http: wttp://.rrrraab//2..1113323
arXiv: 2202.11553
Yichen Wei, Fang Wen, Wangjiang Zhu, and Jian Sun. Geodesic Saliency Using Background Priors.
pp. 14, 2012.
Ross Wightman. PyTorch Image Models, 2019. URL httpptts///gghhbb...mm/////
pytorch-imee image models
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.
1rXiv:2105.1520 [cs], October 2021. URL http:///axx....rg/abb//////////s arXiv: 2105.15203.
Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas Huang. Deep Interactive Object Selection. arXiv:/603. 04042 [es], March 2016. URL http::/aarclv.oorg/aa////// 04042 arXiv: 1603.04042.
Yung- Yu Chuang. B. Curless, D.H. Salesin, and R. Szeliski. A Bayesian approach digital matting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vi- sion and Pattern Recognition. CVPR 2001, volume 2, pp. 1--264-1-227, Kauai, HI, USA, 2001. IEEE Comput. Soc. ISBN 978-0-7695-12722 1272-3. doi: 10. 10.1109/CVP.22001999999 URL httpp://eeexplooe
ieee.
Chuyu Zhang, Chuanyang Hu, Yongfei Liu, and Xuming He. Intention-aware Feature Propagation
Network for Interactive Segmentation. arXiv:2203. 05145 [cs], March 2022. URL http://
arxiv. abs 2203. 05145
arXiv: 2203.05145.
Richard Zhang, Phillip Isola, and Alexci A. Efros. Colorful Image Colorization, October 2016. URL
http://arxiv: org/ http://..ss/......... 001.663..11111 08511 arXiv:1603.080 [es].
Shiyin Zhang, Jun Hao Liew, Yunchao Wei, Shikui Wei, and Yao Zhao. Interactive Object Seg- mentation With Inside-Outside Guidance. 2020 TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12231-12241, Seattle, WA. USA. June 2020. IEEE. ISBN URL httph:ttt///co /etp:///eeeeeepeeee./
978-1-72817-166-5. doi: 10. 1.110/CVV42220022200002222 osse.oogg/occumet/// 91177333
Hengshuang Zhao, Jiaya Jia, and Vladien Koltun. Exploring Self-attention for Image Recognition, April 2020. http://:
org/ abs/2004 13621 arXiv:2004.1362. [cs].
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Scholkopf. Learning with Local and Global Consistency. pp. 8, 2004.
Xiaojin Zhu and Zoubin Ghahramani. Learning from Labeled and Unlabeled Data with URL http::/mig. eng. cam. ac. wtw.:////////pppe///////
Label Propagation, 2002.
100-CALD-02-107
13

---

Under review as a conference paper at ICLR 2023
A BACKBONE ANALYSIS
State-of-the methods in unsupervised segmentation (Siméoni al. 2021; Amir et al. 2021) showed through ablation studies that using keys from the last ViT-B/I6 layer patch represen- tations yielded best results for DINO pretrainings. Here we will see this is true for our task, and if it is also true for other pretrainings. Our analysis all conducted on Pascal VOC 2007 [Everingham et al. 2010).
Table 1: mloU analysis of the different ViT-B/16 backbones for each feature of each layer. For each layer L E [0, 11]. each row corresponds to keys, queries, and values. Four supervised methods (vanilla ViT and ViT-21k pretrained ImageNet-Ik, and -21k respectively; Dosovitskiy iesociiielly.Doovvitskkkyyy
2021 CLIP, pretrained on OpenAl custom dataset: Radford et al. 2021] Segmenter (Strudel et al. 2021 trained on ADE20K), Four self-supervised methods pretrained on ImageNet- (DINO: Caron MAE: He et al. 2021] MoCo-v3; He et al. 2020; MSN: Assran et al. (022). For each column,
2021
Green corresponds to best results, blue to the following five best, and red to the three worst.
Layer (k, K,9,u)
Supervised
Self-supervised
VIT
ViT-21k
CLIP
Segmenter
DINO
MAE
MoCo-v3
MSN
73.2
73.4
73.7
71.3
73.7
74.0
61.2
71.1
70.3
72.3
54.2
67.6
40.7
45.9
46.0
48.4
74.1
73.0
74.5
73.3
73.2
73.5
57.0
74.2
76.0
75.2
77.2
76.3
75.2
75.0
76.1
75.9
75.7
74.8
76.6
76.2
75.3
54.0
75.2
76.8
75.8
75.1
77.0
76.1
77.3
75.4
76.2
77.0
76.5
76.3
75.7
76.5
75.7
77.4
73.1
75.4
2
76.8
76.4
76.6
77.0
76.3
66.3
72.9
76.3
76.3
74.8
78.1
76.9
78.0
77.9
77.4
77.6
75.7
76.3
76.5
76.1
75.7
78.4
74.8
76.0
3
76.8
76.4
77.2
77.1
76.2
72.5
75.1
76.5
76.3
75.8
77.1
77.8
78.0
78.7
77.2
78.8
76.0
75.9
76.7
76.5
77.0
75.2
76.1
77.7
76.9
76.7
76.8
77.5
77.2
75.6
75.8
78.0
76.9
76.7
78.0
78.1
79.3
78.7
77.5
79.1
76.6
76.5
75.0
77.6
77.1
78.8
76.5
78.0
77.3
77.1
76.0
78.6
78.0
76.6
78.4
77.7
77.5
78.1
79.3
79.1
77.7
78.4
79.8
76.7
76.7
77.6
78.1
78.2
74.6
77.8
78.5
6
77.1
77.0
78.1
78.5
79.3
74.9
78.1
79.3
78.2
78.0
78.6
79.7
79.3
79.4
79.0
79.6
77.6
77.5
78.7
79.0
79.0
78.0
79.9
7
78.0
77.9
78.8
79.8
80.0
77.8
78.7
80.0
78.5
78.4
79.3
80.0
79.6
78.4
79.2
80.0
78.8
78.6
79.0
80.2
80.4
78.8
79.1
80.5
78.9
78.6
79.0
80.6
80.8
80.0
79.8
80.7
79.2
79.0
79.0
80.5
79.9
79.6
79.8
79.8
79.3
78.9
79.5
81.0
80.8
80.0
80.1
80.7
79.3
78.9
79.3
81.1
81.1
80.1
80.7
81.0
79.1
79.1
78.8
80.8
80.0
80.3
79.9
79.9
79.9
79.4
79.8
81.4
80.9
79.9
80.8
80.6
10
79.5
78.9
79.4
81.5
81.3
80.3
81.2
80.9
79.3
79.2
78.5
81.0
80.0
79.8
80.3
79.8
79.2
78.9
80.4
81.4
80.8
80.3
80.8
81.0
11
78.6
78.2
80.3
81.3
81.3
80.2
80.7
81.2
78.8
78.6
78.9
81.3
80.1
79.8
80.4
80.0
14

---

Under review as a conference paper at ICLR 2023
Table shows results for ViT-B/16 architectures, across all layers, and for all features: keys (k), queries (q), and values (v). We highlight, for a given pre-training, in green best results, in blue the following five best results, and in red the three worst ones. Overall, results are very similar across all pre-trainings.
Although the variation very small, each pre- pre-training manages to beat vanilla ViTs, Besides, keys from the 11th layer are not necessarily the best representations. Even if they achieve very good results, other ones such as queries from the 10th layer are on par.
B GRAPH ANALYSIS
In this section we study the performance impact when we use different forms of graphs. We will test three different forms of graphs: RBF, KNN, and similarity (see Section 3). We will also add color information to the graph, as in Aiells-KKrriae et al. (2022)
A A feat + AAcolor
where feat is the adjacency matrix for the graph defined using VIT features, À is hyperparameter, and Acolor a KNN graph defined with color information. The similarity graph has hyperparam- tune VOC07, and find that
eter 7 that we will use this value for the À analysis. We find that mixing the features graph with the color graph, in Melas- Melas-Kyriazi al. (2022), only decreased the performance of the algorithm. Thus, conducted all our analysis using the default RBF graph.
0.8 achieves best results (see Figure ( Hence,
80
80
KNW
Similarity
70
mlou
76
50
60
Figure 8: (left) r analysis for similarity graph. (right) À analysis on VOCO7.
C RESULTS PER CATEGORY
Table 2 and shows mloU results per category for COCO-panoptic. As expected, categories that have complex masks, usually with high convex hull over area ratio, achieve the lowest scores, i.e. bicycle. Also, very small objects, that have small ends with sizes below patch area perform very bad, i.e. toothbrush, fork. On the other hand, objects with low convex hull over area ratio achieve better results, i.e. bus, train. Usually, these objects are alone on the image and located at the center with a clear background. The performance of the algorithm is also related to the nature of ImageNet images.
We also analyse the statistics of the user interaction per category. Let be the user interaction over the whole dataset. We report the mean m[I), and standard deviation o(I) percentage of labeled patches by the user. We denote by I+. and I- the correctly, and incorrectly labeled patches re- spectively. Finally, we denote by Tot, the total number of patches per category. We notice that the category with the highest number of incorrectly labeled patches is Fork, which also has one of the lowest average number of patches, and achieves one of the lowest mloU. This can be explained as fork labels are usually very thin, and do not cover entire patches.
15

---

Under review as a conference paper at ICLR 2023
Table 2: Pascal VOCO7 results, and number of interactions per category. Let I be the user interac- tion over the whole dataset. We define mT) as the mean percentage I, and o as its standard deviation. We denote by m(I+) (m((")) the mean over the correctly (incorrectly) labeled interac- tions. The mean, and standard deviation, of the total number of patches are respectively denoted by m(Tot,, (Tot). We use patches of size 16 16.
Category
mloU
m(I*)
m(I)
o(I)
m(Tot)
o (Tot)
Background
90.3
7%
0.0%
3%
3001
1182
person
66.4
19%
0.6%
10%
289
432
bicycle
58.2
19%
2.1%
12%
341
684
62.3
19%
0.3%
8%
177
348
motorcycle
71.1
15%
0.3%
8%
516
573
airplane
74.7
21%
1.8%
16%
316
312
bus
77.2
16%
0.4%
11%
744
786
train
79.3
12%
0.1%
6%
740
718
truck
70.3
16%
0.3%
8%
405
609
boat
61.8
20%
0.6%
11%
233
271
traffic light
56.8
22%
0.3%
9%
116
199
fire hydrant
83.5
15%
0.0%
7%
522
453
stop sign
88.8
13%
0.0%
9%
639
638
parking meter
81.0
14%
0.2%
8%
381
328
bench
64.3
17%
0.8%
14%
428
632
bird
67.8
21%
0.5%
12%
177
247
cat
84.4
12%
0.0%
7%
881
794
dog
80.3
14%
0.1%
8%
649
633
horse
65.6
18%
0.1%
7%
289
320
sheep
64.4
18%
0.5%
8%
184
286
cow
68.0
18%
0.3%
8%
232
255
elephant
70.2
16%
0.0%
8%
471
572
bear
84.0
12%
0.0%
7%
1163
1219
zebra
62.9
17%
0.2%
7%
394
522
giraffe
67.7
23%
1.1%
10%
280
321
backpack
59.7
21%
1.9%
15%
116
184
umbrella
75.0
18%
0.5%
9%
302
466
handbag
65.1
19%
1.1%
11%
75
53
tie
71.4
20%
1.1%
15%
171
124
suitcase
72.5
16%
0.2%
8%
257
390
frisbee
87.1
27%
0.0%
16%
81
73
skis
37.7
33%
9.9%
18%
53
40
snowboard
69.5
24%
1.4%
10%
176
190
sports ball
75.2
17%
0.0%
12%
47
kite
67.7
21%
1.0%
10%
96
71
baseball bat
58.4
30%
3.6%
10%
76
59
baseball glove
74.3
18%
0.1%
8%
136
299
skateboard
62.9
26%
4.2%
16%
153
292
surfboard
67.5
23%
1.7%
17%
150
267
tennis racket
71.9
22%
0.6%
10%
101
115
bottle
59.2
22%
0.6%
10%
117
162
16

---

Under review as a conference paper at ICLR 2023
Table 3: Same as Table for remaining categories.
Category
mloU
m(I+)
m(I
m
(Tot)
wine glass
59.6
17%
0.2%
7%
105
83
cup
68.8
18%
0.4%
12%
108
99
fork
48.4
26%
5.2%
15%
101
129
knife
51.8
33%
4.1%
17%
80
83
spoon
52.8
26%
3.3%
15%
71
59
bowl
74.4
16%
0.3%
9%
583
849
banana
62.0
16%
0.1%
8%
231
112
apple
63.7
15%
0.0%
7%
194
221
sandwich
75.2
11%
0.0%
7%
364
382
orange
65.4
14%
0.0%
8%
189
312
broccoli
58.5
18%
0.0%
7%
139
208
carrot
61.6
23%
0.1%
9%
140
200
hot dog
69.6
18%
0.1%
11%
353
170
pizza
80.4
13%
0.0%
8%
789
965
donut
56.8
23%
0.3%
10%
68
99
cake
74.3
16%
0.1%
8%
411
539
chair
55.3
20%
1.2%
14%
125
209
couch
77.1
14%
0.1%
7%
419
526
potted plant
59.2
18%
0.7%
8%
121
181
bed
78.8
9%
0.4%
5%
1137
853
dining table
72.7
13%
0.2%
8%
868
985
toilet
70.2
15%
0.2%
6%
329
346
76.8
15%
0.1%
8%
199
660
laptop
70.6
14%
0.4%
9%
340
378
mouse
65.5
15%
0.0%
8%
79
67
remote
59.1
20%
1.6%
14%
78
65
keyboard
75.3
15%
0.0%
7%
277
353
cell phone
72.1
17%
0.2%
12%
341
538
microwave
72.8
19%
0.0%
6%
127
130
oven
69.2
17%
0.7%
10%
290
311
toaster
65.0
10%
0.0%
6%
644
592
sink
69.5
18%
0.3%
9%
198
211
refrigerator
75.6
12%
0.1%
6%
645
914
book
60.7
19%
0.3%
9%
116
148
clock
66.7
18%
1.2%
13%
157
215
vase
60.9
19%
0.0%
11%
201
312
scissors
56.7
15%
0.1%
10%
666
869
teddy bear
78.3
14%
0.0%
8%
537
628
hair drier
71.0
19%
0.0%
8%
200
159
toothbrush
47.9
23%
1.2%
9%
48
6
17

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL RESULTS
In this section we present additional results for COCO-panoptic. Figure results when using only fraction of labels. Figure I0 shows results when we add noise to object interactions. Figure
11 shows additional results using one patch only. Figure [2 shows failure cases. Figure 13 to 16
show additional random results for the default configuration.
Figure 9: Analysis SALT robustness against using a fraction of the handmade labels using MAE- B/16 keys from 1 11th layer. From left to right, original COCO-panoptic image (00000001 1760) and ground truth labels, the predicted labels and interactions for p [0.1, 0.99].
18

---

Under review as a conference paper at ICLR 2023
Figure 10: Analysis of SALT robustness against noise. From left to right, original COCO- panoptic image (000000011766) 1760) and ground truth labels, the predicted labels and interactions for p E [0.05, 0.5].
19

---

Under review as a conference paper at ICLR 2023
W
R
7
Figure 11: Same Figure
20

---

Under review as a conference paper at ICLR 2023
Figure 12: Failure cases for COCO -on0-ppan.pti. Same as Figure
21

---

Under review as a conference paper at ICLR 2023
Oio
Figure 13: Same Figure
22

---

conterunee a as
review Unnder
paperr at (CLL 2023
AvA
STOP
Figure 14: Same Figure
23

---

Under review as a conference paper at ICLR 2023
Figure 15: Same Figure
24

---

Under review as a conference paper at ICLR 2023
Figure 16: Same as Figure
25


Reviews:
[{"review_id": "pmexb-WoKd", "reviewer": "Reviewer_Uyni", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an interactive segmentation framework that leverages a pre-trained ViT model to perform semi-supervised segmentation tasks on otherwise challenging images. Due to the nature of the proposed framework and ambiguity in evaluation for interactive segmentation framework, no baseline is provided or compared against. The author provides some ablation study within the proposed framework.", "summary_of_the_review": "It is an interesting application of ViT in semi-supervised interactive segmentation framework. Comprehensive hyper-parameter tuning and experimentation with extreme settings are provided within the proposed framework. To facilitate reproducibility, I strongly recommend the author publish their code upon acceptance.", "strength_and_weaknesses": "Strength\n + An interesting application of ViT in semi-supervised interactive segmentation framework\n + Comprehensive hyper-parameter tuning and experimentation with extreme settings\n\nWeaknesses\n + Although already stated in the paper with understandable reasons, it's still worth pointing out that this work is lacking comparison with existing works and alternative solutions.\n + No evidence is provided as far as I know to substantiate the claim that \"representation of ViT are not able to segment images, with several classes and object instance, in an unsupervised fashion yet\". I believe the claim is true, but proof is needed in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c3H57XgcJ-6", "reviewer": "Reviewer_yrjR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a semi-supervised segmentation with self-supervised attention layers for interactive multi-instance segmentation. Remarkable experiment results are obtained when evaluating on Pascal and COCO-panoptic data with very noisy and sparse labels. ", "summary_of_the_review": "Based on the weaknesses mentioned above, my current decision is reject.", "strength_and_weaknesses": "Strength\n\n1.\tMulti-instance segmentation with less label and human interaction is interesting and practically useful research topic.\n\nWeaknesses\n\n1.\tThe paper is not organized and written well so that it is hard to follow and understand the idea.\n2.\tThe motivation is clearly introduced, thus the underlying principle of proposing the SALT is not understandable.\n3.\tThe experiments are not convincing, no comparison with other state-of-the-art methods. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KukEMEqjf1", "reviewer": "Reviewer_JhQx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an interactive algorithm for multi-class/multi-instance segmentation based on the vision features obtained from self-supervised pre-training. Previously, such vision features, e.g. DINO features, have been used for unsupervised segmentation. This work extends previous work to the interactive segmentation of multiple instances. Similar to prior work, a graph is constructed based on the features learned in the pre-training stage. Then, class-differentiated scribbles from users are used to initialize partial labels. Finally, a label propagation algorithm (Zhou et al., 2004) is adopted to propagate partial labels to the entire image.  Experiments on VOC 2007 and COCO-panoptic show that the proposed method can achieve reasonable performance and the method is quite robust against noise. ", "summary_of_the_review": "This paper presents a new way to do interactive multi-instance segmentation. However, considering the prior arts on DINO-based unsupervised segmentation and interactive segmentation methods, and the fact that the key components including graph construction and label propagation are not new, the novelty of the work is not high. ", "strength_and_weaknesses": "Strength:\n- The paper shows that pre-trained vision features can be used for semi-supervised multi-class/multi-instance segmentation, where the supervision signals can be as simple as scribbles or even sparse points, which are obtained with low manual cost. \n- The work investigated multiple choices in the system, including the pre-train method to obtain vision features (both supervised and self-supervised), the metrics to build the graph (similarity, RBF, and KNN), and the quality of the supervision signal (partial missing and noisy). \n\nWeakness:\n- The novelty of the work is not high, as it is a simple extension to existing methods (unsupervised segmentation using pre-trained vision features), and there is no big challenge in this extension. The work follows previous work in graph construction and label propagation. \n- The work is not compared with other work. One reason might be that the problem setting is too specific. But how does this setting compare with other interactive segmentation settings?\n- The authors simply ignore small objects, but the segmentation accuracy of small objects is important to the segmentation problem. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fwr4y6DSH0", "reviewer": "Reviewer_KmRX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors try to address the multi-instance interactive segmentation task instead of salient interactive segmentation via the Label Propagation algorithm with patch feature representation from Transformers.\n\nIn particular, the authors utilize keys from the last ViT-B layer for feature representation which is later leveraged to construct a graph. A user is assigned to draw scribbles for different objects in an image and apply the Label Propagation algorithm to propagate information through the graph and the remaining nodes. \t\nThe main contribution of this paper is that this work evaluates the representation power of self-supervised ViTs for interactive segmentation tasks.  \n\nExperiments illustrate the effectiveness of the last ViT-B layer with different pre-trainings and demonstrate the robustness against noise and using different ratios of interactive annotations. \n", "summary_of_the_review": "The main contribution of this work is utilizing feature representation of ViTs and Label Propagation for interactive segmentation tasks. Although experiments comprehensively analyze the effectiveness of different pre-trainings as well as the performance of proposed methods in extreme conditions, the contribution of this work is minor and does not impress me a lot. Moreover, experiments do not compare with current methods in interactive segmentation tasks which can not prove outstanding results. \n", "strength_and_weaknesses": "The main contribution of this work is applying feature representation of self-supervised ViTs and Label Propagation for interactive segmentation tasks. The authors also assess the performance of their algorithm in extreme circumstances with very sparse information on the objects’ position. In addition, the methodology of annotating datasets in an interactive way is carefully described.\n\nThis paper is presented in a clear, coherent manner.  Each module is explained thoroughly and experiments provide detailed information. \n\nHowever, the contributions are not much impressive and not considerable as this paper is just a combination of previous modules. For instance, the authors utilize the last ViT-B layer for building a graph and also follow the prior work (Simeoni et al. 2021).  Moreover, the authors reuse the Label Propagation without significant modification and contribution. \n\nExperiments lack comparison with other methods in interactive object segmentation. For instance,  f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation - CVPR 2020 or EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow - ICCV 2021. It is difficult to convince that this proposal stands out as a state-of-the-art method.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject