PaintSeg: Painting Pixels for Training-free Segmentation
Authors: Xiang Li
Chung-Ching Lin
Yinpeng Chen
Zicheng Liu
Jinglu Wang
Rita Singh
Bhiksha Raj
Venue: NeurIPS 2023 poster (2023)
Keywords: Prompt-guided Segmentation
Generative models
Training-free
TL;DR: The paper introduces PaintSeg, a new unsupervised method for segmenting objects without any training.

Abstract: The paper introduces PaintSeg, a new unsupervised method for segmenting objects without any training. We propose an adversarial masked contrastive painting (AMCP) process, which creates a contrast between the original image and a painted image in which a masked area is painted using off-the-shelf generative models. During the painting process, inpainting and outpainting are alternated, with the former masking the foreground and filling in the background, and the latter masking the background while recovering the missing part of the foreground object. Inpainting and outpainting, also referred to as I-step and O-step, allow our method to gradually advance the target segmentation mask toward the ground truth without supervision or training. PaintSeg can be configured to work with a variety of prompts, e.g. coarse masks, boxes, scribbles, and points. Our experimental results demonstrate that PaintSeg outperforms existing approaches in coarse mask-prompt, box-prompt, and point-prompt segmentation tasks, providing a training-free solution suitable for unsupervised segmentation. Code: https://github.com/lxa9867/PaintSeg.

PaintSeg: Training-free Segmentation via Painting
Xiang Li', Chung-Ching Lin", Yinpeng Chen', Zicheng Liu", Jinglu Wang?, Rita Singh', Bhiksha Raj CMU, Microsoff,, "MBZUAI x1 68 andrew. cmu. edu
Abstract
The paper introduces PaintSeg, a new unsupervised method for segmenting ob- jects without any training. We propose an adversarial masked contrastive paint- ing (AMCP) process, which creates contrast between the original image and a painted image in which a masked area is painted using off-the-shelf generative models. During the painting process, inpainting and outpainting are alternated, with the former masking the foreground and filling in the background, and the latter masking the background while recovering the missing part of the foreground object. Inpainting and outpainting, also referred to I-step and O-step, allow our method to gradually advance the target segmentation mask toward the ground truth without supervision training. PaintSeg a variety of prompts, e.g. coarse masks, boxes, scribbles, and points. Our ex- perimental results demonstrate that PaintSeg outperforms existing approaches box-prompe and viding training-free solution suitable for unsupervised segmentation. Code: https:/ peteps:/giiie //tp::/gggttt/tt/ com xa 92
be configured to work with
in coarse
segmentation tasks, pro-
1 Introduction
With deep learning advancements, significant progress has been made in the field of image generation and segmentation in recent years. A particular generative model, the denoising diffusion probabilistic model (DDPM), has demonstrated outstanding performance
variety of generative tasks, such as
[21 20 79). Similar developments have occurred in the field of object segmentation, such as the strong zero-shot capability and excellent
image inpainting [49 16 (34 and text-to-image synthesis
segmentation quality demonstrated by SAM
(30]
Image generation and segmentation can be mutually beneficial. Segmentation has been shown to be critical technique in improving the realism and stability of generative models by providing pixel-level guidance during the synthesis process 77] 29]. Interesting note the fact that the relationship between segmentation and generative models does not appear to be solely one-sided. Generative models learning to "paint" objects actually know where the painted object is. The emergence of unsupervised image segmentation methods utilizing generative adversarial networks (GANs) has using generative models.
produced a line of methods that can segment objects in images 44 These methods work on the assumption that object appearance and location can be perturbed without compromising scene realism. By using the GAN architecture to discriminate between perturbed and real images, these methods can achieve effective object segmentation. Moreover, follow-up work [61 develops an approach to leverage pre-trained GAN by identifying "segmenting" direction the latent space to discriminate object shapes.
In this paper, we present PaintSeg, an approach for unsupervised image segmentation that leverages off-the-shelf generative models. Unlike previous methods [6]] 4 that require training on top of these models, PaintSeg introduces novel, training free segmentation approach that relies on an adversarial masked contrastive painting (AMCP) process. The AMCP process creates contrast
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

I-Step
Contrast
O-Step
Contrasi
Mask
Image
AMCP
Mask
Paint
Mask
Paint
Masked Image
Painted Image
Masked Image
Painted Image
Image
Init. BG Mask
FG Mask
BG Mask
Figure 1: Illustration of adversarial masked contrastive painting (AMCP), Given input image and an initial mask, AMCP leverages alternating I-step and O-step to gradually refine the segmentation mask until converges the ground truth. Both steps share the same mask, paint, and contrast operations. The updated mask in each step is achieved by binarizing the contrastive difference between the original and painted images.
between the original image and painted image by alternating between inpainting and outpainting, with the former filling in the background and masking the foreground, and the latter retrieving the missing part of the object while masking the background and portion of the foreground.
Both steps, as shown in Fig. [ share the same operations while taking input from background and foreground masks, correspondingly. In the I-step, the object region is removed from the painted image, creating a significant contrast with the original image. Conversely, the O-step, the background region exhibits remarkable difference between the original and painted image. The foreground or background mask can be obtained by binarizing the contrastive difference each step.
Although either I-step or O-step is capable of discriminating objects, the single-step method is less robust. The I-step involves segmenting objects based on background consistency without taking into account object information. As a result, the segmentation may be imperfect if the object part resembles the background. Similarly, the O-step, only the object shape prior is utilized, resulting in lack of background knowledge. This problem is addressed by introducing adversarial mask updating, which I-steps and O-steps are alternated. During I-step, we only shrink the object mask to cut off background false positives, while during O-step, expand it to link up foreground false negatives. Thereby, even errors occur during the iteration AMCP, they will be corrected next step without degradation. With the adversarial mask updating, the target mask can be gradually advanced to the ground truth.
With the robustness of AMCP, PaintSeg can deal with inaccurate initial masks and adapt to various visual prompts, such as coarse masks, bounding boxes, scribbles, and points. Compared to the recently published successes in image object segmentation study, our main contributions are as follows:
We propose PaintSeg, training- approach segmenting image objects based heterogeneous visual cues. The method provides direct bridge between generative models and segmentation.
We introduce adversarial masked contrastive painting (AMCP), consisting of alternating I-step and O-step, to robustly segment objects.
We conduct extensive experiments for analysis and comparisons on seven different image segmentation datasets, the results of which show the superiority and generalization ability of our methods.
2 Related Works
2.1 Unsupervised Image Segmentation
Unsupervised methods for image segmentation are extensively investigated with the advancements in self- self-supervised. DINO [ provides a self-supervieed approach to explicitly bring out underlying
semantic segmentation of images using a Vision Transformer (ViT) [17. Based on DINO, LOST [56].
Deep Spectral Methods 44 and TokenCut (66) leverage self-supervised ViT features and propose to

---

segment objects using NCut [52] Subsequently, [57 55 introduce a second-stage training approach incorporates background similarity as an additional refinement factor, while SelfMask [S utilizes an ensemble of features 7 12 to enhance
to further improve the segmentation quality. Found [57]
image representation. CutLER (65] enables multiple objects discovery capability by iteratively cutting
objects with NCut and introduces a more powerful second-stage training. FreeSOLO [4] generates coarse masks with correlation maps that are then ranked and filtered by a "maskness" score. Another line of unsupervised methods learns to generate realistic image by combining foreground, background and a mask [5] byproduct.
25 and then the object segmentor can be obtained as a
2.2 Prompt-guided Segmentation
Prompt-guided segmentation aims to segment objects assigned by prompts, e.g.. mask, box, scribble and point. Semi-supervised video object segmentation (VOS) [1 68 38 aiming at segmenting object
masks across frames given the first frame mask, typical mask- prompt task. The mainstream of
VOS methods 772 [73 constructs pixel-level correspondence and propagates masks by exploring
matches among adjacent frames. Interactive segmentation (IS) [78, 40, 58, 23 is another line of
prompt-guided segmentation. IS permits users to leverage scribbles and points to assign target objects and segment them. In addition, an interactive correction also featured by IS which introduces additional prompts correct misclassified regions. MIS [33] is a recent work tackling unsupervised IS and proposes a multi- coutii-ganullartty region proposal generation to refine the mask. SAM [30] is a recently introduced zero-shot method for prompt- dataset and a strategy to mitigate the ambiguity of prompt. Beyond visual prompts, objects can also be referred by natural language acoustic prompts. Referring image segmentation (RIS) 22 74
segmentation which introduces a large-scale
and referring video object segmentation (R- (R-V [11 [6 13 35 37 aims to segment objects in image/video referred by linguistic expressions. Audiovisual segmentation [77 36 aims segment sound sources in the given audiovisual clip.
2.3 Conditional Image Generation
Conditional image generation refers to the process of generating images based specific conditions or constraints. In most instances, the condition can be based on class labels, partial images, semantic masks, etc. Cascaded Diffusion Models (26] uses ImageNet class labels condition to generate high-resolution images with a two-stage pipeline of multiple diffusion models. [51] guides diffusion models to produce novel images from low-density regions of the data manifold. Apart from these, CLIP 48 has been widely used in guiding image generation in GANs with text prompts [21] 20 79.. For diffusion models, Semantic Diffusion Guidance [41] investigates unified framework for diffusion-based image generation with language, image, or multi-modal conditions. Dhariwal et al. the gradients of classifier to guide the diffusion with trade-off between diversity and fidelity. Additionally, Ho et al. al.[77) introduce classifier- free guidance in conditional diffusion models by mixing the score estimates of a conditional diffusion model and jointly trained unconditional diffusion model.
14 apply an ablated diffusion model
3 Problem Definition
We tackle the unsupervised prompt-guided image object segmentation task, which aims to predict the object mask M E {0, xuxdii in an image E |x given a visual prompt P e {0, 13 1 The visual prompt can have a format of point, scribble, a bounding box or a coarse mask of the target object P E {Pp points Pserib. Proz. Pmask J. Following the convention, we assume the ground -ouut object mask M must have an overlap with the visual prompt P PMM + 0.
4 Adversarial Masked Contrastive Painting
PaintSeg leverages adversarial masked contrastive painting (AMCP) to gradually refine the initial prompt P to the object mask M. The AMCP approach is composed of alternating I-steps and O-steps, as illustrated in Figure I] During each step, region of the image is masked out based on the previous iteration's 's mask, and the masked region is then repainted and compared to the original image to refine the mask prediction. To improve the segmentation's 's robustness, PaintSeg introduces adversarial
3

---

Mask
Image
Mask
Image
Contrast
Contrast
Mask with FP
Mask with FN
Prompc
Contrustive Potantial
Paint
Contrastive Potential
Paint
FN
(a) I-step given mask with FP
(b) O-step given mask with FN
Figure 2: Illustration of I-step and O-step with initial prompts. (a) We show I-step with box prompt as the initial mask. where the object region has a sigmificant difference between the original and painted images. (b) We show an O-step with scribble prompt the initial mask, where the object region has a small difference between the original and painted images.
mask updating, which helps to ensure that the mask accurately reflects the object's boundaries. The I-step used shrink the object mask by leveraging background consistency, thereby eliminating false-positive regions. On the other hand, the O-step expands the object mask by utilizing object shape consistency link up felse-negative foreground regions.
4.1 Contrastive Painting
We first discuss the rationality of segmenting objects by contrasting painted and original images. Given visual prompt P. the relation between the prompted area and object mask can be categorized into three types: background false positive (Fig. 2 (a)). foreground false negative (Fig. (b)) and a hybrid of both. We tackle the prompt- guided segmentation by separately addressing the background false positive and foreground false negative with I-step and O-step respectively.
We discuss the painted content with different mask situations. To avoid ambiguity, we first denote the generative model taking background and foreground as conditions as inpainting model Φ(.) outpainting model vb(-) respectively. We consider the prompted area as the initial mask Mo = P. When the initial mask has false positives, i.e., M Mo. as shown Fig. (a). the inpainted content 2
tends to complete the background based on the background consistency. In this way, the inpainted pixels inside the object will have significant difference compared when the initial mask has false negatives, i.e., Mo C M, as shown in Fig. 2 (b), the outpainted content tends to complete the partial object leading to a low difference with the original image inside object region. We notice that I-step address background false-pose possiivve and O-step address foreground nalse-negative. By alternating conducting I-step and O-step, we can leverage both foreground and background consistency and address more complicated cases.
original image. In contrast,
4.2 Contrastive Potential
Given the image I and mask Mr. we define contrastive potential & measure the region relations, which contains three terms
Apaina o paint + Acotor D Pootor + Aprompe 4prompepprompee promptt.
(1)
We introduce box region B that encloses the foreground and define the Ppeint term as the distance between the painted and the original image. Specifically, Ppeint B o (E(I) E (lpaini))2, where 123x HI xW + CC' (CxxHxx HxW is function that projects the image to high-dimensional space. denotes the Hadamard product.
The &coler term measures the pixel-level color similarity inside and outside the mask M,. To compute it,
the output of the conditional random field algorithm [II) and define Pcotor C(M, B, I o B), where C is a function that takes a mask and an image as inputs and outputs the probability of whether pixel should belong the masked region.
To further incorporate prompt information, we introduce prompt priors + prompt to the contrastive potential for box. scribble, and point prompts. Let us denote [xt, m)] as the coordinates of the 1-th

---

point in the prompt area. The prompt prior is defined as follows:
prompt [i = max (Iz, yr) [.
(2)
Specifically, for the box prompt, we only take the center point of the box into account. The prompt priors are designed to leverage the positional information of the prompts better locate the target object. By taking the maximum value of the Gaussian function over all points in the prompt area, T prompp captures the overall strength of the prompt signal.
where G is two-dimensional Gaussian function, and G|d, = exp( exp("==z)) "
4.3 Adversarial 1/O-Step
As shown in Fig. [ I-step and O-step share the same mask, paint, and contrast processes while the input mask in I-step is the background mask and, in O-step, the foreground mask. Given the original image I and an input mask M, from the f-th step of AMCP (assuming M, is a background mask thus + 1-th step is an I-step), we first filter out the masked region by o M2 and then paint the image paintt = oi o Me). As discussed in Section [4.1] the foreground region will have a significant difference between painted and original images. We obtain the updated mask M++1 by k-means
clustering XC()) over the contrastive potential Φ. Let us denote μ4 and SN E {0, 1) 1 as the average value of all samples in the k-th cluster and its corresponding identity map (Sx [i.] 1 pixel [i.j] belongs to center & else 0). The updated mask can be found by
M++1 = Sk-, k- arg max pkk &
The updated mask M1+1 is a foreground mask thus the next step will be an O-step. Similarly, we paint the image by [paint (b(I M, Here, the difference in background area will have significant difference between painted and original images. Thereby, the updated mask M242 from O-step can be computed using the same rule as Eq. (3) which leads background mask. By updating the mask by Eq. (3). we notice that when the input mask M, is a background mask, then the output mask will be a foreground mask and vice versa. Thereby, the alternating I-step and O-step can be automatically achieved.
As discussed in Section 4.1 I-step is advantageous for reducing false positives in the background, whereas O-step is beneficial for reducing false negatives in the foreground. Specifically, the updated mask is configured to only cut off pixels in the I-step, and to only link up pixels in the O-step. Let M;* and M, as the dilated and eroded masks of Mc. We constrain to only update the regions near the bregrrund-backgrouun boundary. In this way, the updating rule for AMCP can be rewritten
Sk- o A + M. -1-48 AT),
I-step
Mc+1
k
= arg max pk
(4)
Sk- A + M, 0 (1 - A+). A1-A+).
O-step
where A = M, M, and A = M M, are the inner and outer neighbors of Mp. Through the adversarial alternation of I-steps and O-steps. AMCP can handle more complex cases involving both false positives and false negatives. Due to the randomness inherent in generative painting, we paint the image N times in each step, and use the averaged mask as an output.
4.4 Discussion
In this section, we introduce the mathematical formulation of AMCP. Mathematically. an image can be represented masked combination of foreground image Ir and background image I
1 IF o M + IB o M, M (0, #xxx1
(5)
M is foreground mask. .7 = - M. An inpainting model ob-]] is defined to generate pixels inside the mask given the pixcls outside the mask as a condition. Similarly, an outpainting model +1-]] predicts pixels outside the mask given the pixels inside the mask as a condition. In our method, we aim to find a M that maximizes
arg max [ A-
Hll M) o 11 IIo w M A+
(6)
I-Step
O-Step
5

---

Method
Training
DUTS-TE 62
ECSSD 54
Compared to methods with training
SelfMask
55
CVPRW12
62.6
78.1
SellMask
56
CVPRW22 BS
66.0
81.8
FOUND
CVPR2S
63.7
79.3
FOUND
CVPR2s BS
66.3
80.5
PaintSeg
67.0
80.6
Compared to methods without training
Melas Melas-Kyriazi et
4
$2.8
71.3
LOST
56
RMVC21
51.8
65.4
LOST
56
BMVC21 + BB
57.2
72.3
DSS
CVPR22
51.4
73.3
TokenCut
666
CVPR22
57.6
71.2
TokenCut 66 CVPR22 BS
2
62.4
77.2
SeltMaski 55 CVPRW22
46.6
64.6
FOUNDf [57 CVPR23
71.7
PaintSeg
67.0
80.6
Table 1: Qantitative results of coarse mask-promp prompted segmentation on DUTS-TE and ECSSD. PaintSeg utilizes the coarse mask generated by unsupervised TokenCut [66] as prompt. BS denotes the application of the post-processing bilateral solver on the generated masks and the column 'Learning' specifies which methods have a training step. The best result per section is highlighted bold. The second best result for each section is underlined. f indicates the first-stage pseudo mask obtained without training.
The first term aims to maximize the difference between the original image I and the inpainted image 6(I M) in the inner neighbor A which corresponds the I-step in AMCP. The second term aims to maximize the difference between the original image I and the outpainted image vi[ M] in the outer neighbor A corresponding to the O-step.
In each step, our mask, paint, and contrast operations can be considered as an expectation- maximization-lii (EM-like) process with the latent variable of 4point maximize Eq. (6). On one hand, the Ipainn is estimated by the mask and paint operations where the conditional probability p(1point\// /,, .M) characterized by the generative painting models (expectation step). On the other hand, the predicted mask M can be updated by maximizing the contrastive potential & (maximiza- tion step). Since the EM algorithm is sensitive the initial value, solely updating with I-step or O-step cannot achieve robust performance. With the alternating I-step and O-step, we introduce an adversarial updating process which leads a more robust mask estimation.
5
Experiment
5.1
Datasets
For mask-prompt segmentation, we evaluate on DUTS-TE 63 and ECSSD 53
DUTS-TE contains
5,019 images selected from the SUN dataset [67 and ImageNet test set [3 ECSSD 53 contains 1,000 images that were selected to represent complex scenes. For box-prompt segmentation, we evaluate on PASCAL VOC [19 val set and COCO 39 MVAL datasets. COCO Val contains 800 object instances from the validation set with 10 images from each of the 80 categories. For point-prompt segmentation, we use three datasets including GrabCut (50) which contains 50 images and corresponding segmentation masks that delineate foreground object; Berkeley [22 which contains 96 images with 100 instances with more difficulty than GrabCut and DAVIS [47] which is
video dataset and 10% of the annotated frames are randomly selected, yielding 345 images that are used in the evaluation
5.2 Experimental Setup
Evaluation metrics. In accordance with previous methods 30 66 quality using intersection over union (IoU).
evaluate segmentation
Implementation details. We leverage the inpainting models trained with latent-diffusion pipeline 49 as our & and s. We set the diffusion iterations 50. We leverage DINO [8] pretrained VIT-S/8

---

O-step. We average N=5 painted images to obtain the updated mask for each step. The 0s and dy are set to TO of are the Training Supervision GrabCut
Method
Berkeley
DAVIS
Compared to methods with training
DIOS 69 OVPRI6
64.0
66.0
57.8
RITM 558
ICIP22
81.0 OO
77.7
66.0
MIS 33
76.2
63.2
53.3
PaintSeg
84,4
70.0
69.4
Compared methods without training
Random Walk
22
TPAM106
25.7
26.2
<20
GrowCut 60
GraphiConti
26.7
26.2
GraphCut ICCV01
41.8
33.9
<20
PaintSeg
84.4
70.0
69.4
Table 2: Qantitative comparison of
segmentation on GrabCut, Berkeley, and
DAVIS The point prompt is given as the centroid of each object.
Method
Training Supervision PASCAL VOC
M
Compared methody with training
Mask-RCNN 24
ICCV17
V
73.2
79.4
CuiLER 665
CVPR23
63.5
74.8
PaintSeg
59.7
69.6
Compared to methods without training
TokenCut 666
CVPR22
30.2
34.7
PaintSeg
59.7
69.6
Table 3: Qantitative comparison of bux- box-prompted segmentation on PASCAL VOC and COCO MVal.
17 as our E. We use [31 our C(-) to calculate ecolor- If no specification, for all experiments, the masked contrastive painting starts from the I-step and updates for steps. We set the number of cluster centers to 3 in the first three steps for point, box and scribble prompts otherwise 2. We set Apoint = 0.8, Aentor = 0.2 and Aprampt = 0.2 I-step and Aprompt = -0.2
the width and height of the bounding box of the current stage mask respectively. + and
neighbors 32 pixels outside and inside the object boundary. We leverage dilation and erosion to filter out sparse points for each iteration. The kernel size set to For the mask and box prompts, we set the prompt as the initial mask. For the point and scribble prompts, set the entire image as the initial masked region. The images are padded to 512 x 512 to fit generative inpainting model.
5.3 Main Results
Coarse mask prompt. Since the usage of the ground-truth coarse mask as a prompt is rare, we evaluate PaintSeg on two unsupervised salient object detection benchmarks and leverage the coarse mask generated from TokenCut [66] as our prompt. As shown in Table PaintSeg achieves encouraging performance that is even comparable with training-basee methods. Under the training- free setting, PaintSeg significantly outperforms previous methods by margin of 4.6 loU on DUTS-TE and 3.4 loU on ECSSD. We attribute the performance improvement the error correction capability of PaintSeg. With alternating between I-step and O-step, the proposed PanintSeg can handle noisy prompts effectively. The robustness of PaintSeg will be discussed in more detail Section 5.4
Point prompt. As shown in Table 2 we compare our method with tate-offthe-as point prompt segmentation approaches. PaintSeg consistently outperforms the training- methods. Even compared to training-based methods with ground truth supervision, PaintSeg still achieves the best performance on GrabCut and DAVIS datasets. MIS 33 is an unsupervised approach equipped with second-stage training. We notice that our method can significantly outperform it in terms of loU, with improvements of 8.2, 6.8, and 16.1 on GrabCut, Berkeley, and DAVIS correspondingly.
Box prompt. Since there is no unsupervised box-prompted segmentation that can be directly compared, we compare the proposed method with several baselines including TokenCut (66), CutLER (55 and MaskRCNN 24 We first cut off the ground truth box region and then run the baselines. As shown in Table 3, when compared with training-based Mask-RCNN and CutLER, PaintSeg shows suboptimal performance, which can be explained by the lack of training handle complex
7

---

A
GT
Point Prompt
RITM
PaintSeg
GT
Mask Prompt
TokenCut + BS
PaintSog
Figure 3: Qualitative results of baselines and our PaintSeg with point and mask prompts. Green point denotes the point prompt. The mask prompt is generated by unsupervised TokenCut [66] BS
represents the bilateral solver
We compare with RITM [58 and TokenCut [66].
Istep
Osstep
AC
loU
Prompt
0% Noise
15% Noise
30% Noise
78.4
Point
60.8
60.4
58.9
/
77.9
Scribble
64.3
63.7
62.7
/
79.5
Box
70.7
70.1
80.6
Coarse Mask
80.6
80.0
79.3
Table 4: Module effectiveness in
Table 5: Prompt robustness. We add random noise to the
AMCP. AC: adversarial constraint for mask updating.
prompt to evaluate the robustness of AMCP. The noise scale is determined by half the length of the object box diagonal.
scenarios. However, as MaskRCNN is trained on 80 COCO object categories, the "unseen" gap remains substantial. PaintSeg provides an alternative solution that is not reliant on training, thus making it more general and capable of handling categories of objects. When compared with unsupervised approaches, our method eclipses TokenCut by large margin on both PASCAL VOC and COCO MVal datascts.
Qualitative results. We visualize the qualitative results with point and coarse mask prompt in Fig. 3 Our visualization depicts comparably reliable results. Comparatively, PaintSeg segments relatively complete object, while baselines miss some parts of it.
5.4
Analyses
Module effectiveness in AMCP. We step by step add proposed modules in AMCP to validate the effectiveness. As shown Table we report the results on ECSSD with coarse-mask prompts. We observe that the missing of either step impacts the performance, as evidenced by the significant drop IoU (compared to alternating I-step and O-step). With the adversarial mask updating constraint, AMCP achieves the best performance of 80.6 loU.
Robustness of AMCP with different prompts. In Table 5 we add noise to the initial prompt by randomly shifting the position investigate the robustness of AMCP. The scale of random noise is determined, w.r.t., half the length of the diagonal of the ground-truth bounding box. We observe that AMCP remains robust and only shows slight performance drop with noise rate of less than 30%. The robust capability can be attributed to 1) the alternating I-step and O-step to leverage both background and object shape consistency, and 2) the adversarial mask updating to tackle the background malse-positives and foreground false-negatives.
Design choices in AMCP. We conduct experiments to ablate the design choices in AMCP and their impacts on the segmentation performance. We first study the effect of cluster center numbers for quantizing contrastive potential. With larger cluster center, AMCP will ignore more ambiguous regions. As shown in Table [a] we notice a cluster center 2 achieves the best performance for mask prompt. After that, we ablate on the AMCP step number in Table 6b] The segmentation performance keeps increasing until reaching a step number of In this way, we choose as our step number. we leverage the diffusion-based generative model, we ablate the iterations for the diffusion process as

---

K
T
Iter
Rate 0.9
IoU 80.6 72.3 61.5
IoU 78.4 79.1 80.6 80.5
IoU 77.3 78.7 80.6
IoU 69.2 72.4 80.6 80.0
(a) Cluster center.
(b) Step number.
Ite. for painting.
(d) Box size for contrasting.
Table 6: Design choices for AMCP. We report the performance with the coarse- mars prompt on
ECSSD. (a) We ablate the cluster center when contrasting. (b) We ablate the step number for AMCP (c) We ablate the diffusion iteration for generative painting. (d) We ablate on the cropped box size when contrasting. The rate denotes the proportion of cropped mask and the box of the current stage object mask.
Input
Step
Step (O)
Step (I)
Step (O)
Step 5 (I)
Mask
Image
Mask
Image
Figure 4: Iterative process of AMCP with box prompt. We inverse the outputted background mask I-step for better comparison. We only visualize the box prompted area.
can impact the image quality, As expected, Table 66 demonstrates that larger iteration number can reach better performance. To filter out irrelevant background regions, we crop box region wrapping the given object mask contrast images. We ablate the box size in Table 6d We notice that a box slightly larger than the bounding box to the given mask can achieve the best performance. explanation for this could be that box tightly enclosing an object will result in a high proportion of object region, which may dominate the features and lead to ambiguity. Properly introducing background can make the extracted features more discriminative and easier for clustering.
Visualization of mask updating. To better illustrate the iterative process of AMCP, shown in Fig. 4 we visualize the averaged mask output (among painted images in each step) for each step with box prompt. As the given mask only contains background false positives, I-step plays major role to cut false-positive backgrounds
AMCP. The mask shrink can also be observed after the O-step which is due the binarization of the averaged mask from the I-step instead of contrastive painting. We observe that the updated masks are gradually closer to the mask of the target object with AMCP.
Conclusion
To conclude, PaintSeg bridges the gap between generative models and segmentation. It designed to provide robust and training-free approach to unsupervised image object segmentation. With the proposed adversarial masked contrastive painting (AMCP) process, PaintSeg creates contrast between the original image and the painted image by alternately applying I-steps (inpainting) and O-steps (outpainting). The alternating I-step and O-step gradually improve the accuracy of the object mask by leveraging consistency in the background and the shape of the object. The competitiveness of our method on seven different image segmentation datasets suggests that PaintSeg can deal with inaccurate initial masks and adapt to various visual prompts, such as coarse masks, bounding boxes,

---

scribbles, and points. An extensive ablation analysis indicates number of key factors and ad advantages of the proposed model, including its design choices and generalizability.
Limitation. In spite of PaintSeg's 's high performance for training-fire image segmentation with heterogeneous visual prompts, it does not possess object discovery capabilities and therefore cannot automatically recognize instance-level masks in an image. Developing discovery capability can be achieved by conducting second-stage training on the segmentation results generated by PaintSeg. which is our future research focus.
References
[1] Ali Athar, Alexander Hermans, Jonathon Luiten, Deva Ramanan, and Bastian Leibe. Tarvis: A unified approach for target-based video segmentation. arXiv preprint (xXiv2330.00265) 2023. The fast bilateral solver. In Computer Ession-ECC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 617-632. Springer, 2016.
[2] Jonathan T Barron and Ben Poole.
[3] Jonathan T. Barron and Ben Poole. The fast bilateral solver. In ECCV, 2016.
[4] Yaniv Benny and Lior Wolf. Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-graine clustering. In Computer -ision-ECCV 2020: 16th Europeun Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pages 514-530. Springer, 2020.
[5] Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. Advances in Neural Information Processing Systems, 32, 2019.
[6] Yuri Y Boykov and M-P Jolly. Interactive graph cuts for optimal boundary & region segmenta- tion of objects in nd images. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume 1, pages 105-112. IEFF, 2001.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEFF/CVF international conference on computer vision, pages 9650-9660, 2021.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[10] Mickaël Chen. Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. Advances in neural information processing systems, 32, 2019.
[11] Weidong Chen, Dexiang Hong. Yuankai Qi, Zhenjun Han, Shuhui Wang, Laiyun Qing, Qing- ming Huang, and Guorong Li. Multi-attention network for compressed video referring object segmentation. In Proceedings of the 30th ACM International Conference Multimedia, pages 4416-4425 2022.
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and patlern recognition, pages 248-255. leee, 2009.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[15] Zihan Ding, Tianrui Hui, Junshi Huang, Xiaoming Wei, Jizhong Han, and Si Liu. Language- bridged spatial-temporal interaction for referring video object segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pages 4974-4973, 2022.
[16] Qiaole Dong, Chenjie Cao, and Yanwei Fu. Incremental transformer structure enhanced image inpainting with masking positional encoding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11358-11368, 2022.
10

---

[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[18] SM Eslami, Nicolas Heess, Theophane Weher, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in neural information processing systems, 29, 2016.
[19] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[20] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
[21] Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. arXiv preprint arXiv:2102.01645, 2021.
[22] Leo Grady. Random walks for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 28(11):1178) 768 1783, 2006.
[23] Yuying Hao, Yi Liu, Zewu Wu, Lin Han, Yizhou Chen, Guowei Chen, Lutao Chu, Shiyu Tang, Zhiliang Yu, Zeyu Chen, et al. Edgeflow: Achieving practical interactive segmentation with edge-guided flow. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1551-1560, 2021.
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[25] Xingzhe He, Bastian Wandt, and Helge Rhodin. Ganseg: Learning to segment by unsupervised hierarchical image generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1225-1233 2022.
[26] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
[27] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint MPXi::2207.12590 2022.
[28] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In European Conference Computer Vision pages 108-124. Springer, 2016.
[29] Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 1125-1134, 2017.
[30] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXivv2334..22443 2023.
[31] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
[32] Hanock Kwak and Byoung-Tak Zhang. Generating images part by part with composite genera- tive adversarial networks. arXiv preprint arXiv: 1607, 1607,05877, 2016.
[33] Kehan Li, Yian Zhao, Zhennan Wang, Zesen Cheng, Peng Jin, Xiangyang Ji, Li Yuan, Chang Liu, and lie Chen. Multi-granularity interaction simulation for unsupervised interactive segmentation. arXiv preprint arXiv:2303.I 13399, 2023.
[34] Xiang Li, Yinpeng Chen, Chung-Ching Lin, Rita Singh, Bhiksha Raj, and Zicheng Liu. Complet- ing visual objects via bridging generation and segmentation. arXiv preprint 10..002310000808 2023.
[35] Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Yan Lu, and Bhiksha Raj. RD 2vos: Robust referning video object segmentation via relational multimodal cycle consistency. arXiv preprint 200.02220. 2022.
11

---

[36] Xiang Li, Jinglu Wang, Xiaohao Xu, Xiulian Peng, Rita Singh, Yan Lu, and Bhiksha Raj. Rethinking audiovisual segmentation with semantic quantization and decomposition. arXiv preprint 1rXiv:2310.00132. 2023.
[37] Xiang Li, Jinglu Wang, Xiaohao Xu, Muqiao Yang, Fan Yang, Yizhou Zhao, Rita Singh, and Bhiksha Raj. Towards noise-tolerant speech-referring video object segmentation: Bridging speech and text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2283-2296, 2023.
[38] Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement. Advances in Neural Information Processing Systems, 33, 2020.
[39] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[40] Zheng Lin, Zheng-Peng Duan, Zhao Zhang, Chun-Le Guo, and Ming-Ming Cheng. Focus- Cut: Diving into a focus view in interactive segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2637-2646, 2022.
[41] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 289-299, 2023.
[42] Kevin McGuinness and Noel E O' connor. A comparative evaluation of interactive segmentation algorithms. Pottern Recognition, 43(2):434 444, 2010.
[43] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an unsu- pervised image segmenter in each of your deep generative models. CoRR, abs/2105.08127, 2021.
[44] Luke Melas KyriazK, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral meth- ods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8364-8375, 2022.
[45] Luke Melas- Kyrrazi,, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral meth- ods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022.
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2304.0::00000193 2023.
[47] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Warkine-Hornung A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 00020, 2021.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[50] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG), 23(3)309--14, 2004.
[51] Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 11492-11501, 2022.
[52] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888- 9058 2000.
12

---

[53] Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierarchical image saliency detection on extended cssd. IEEE transactions on pattern analysis and machine intelligence, 38(4):717-729, 2015.
[54] Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierarchical image saliency detection extended CSSD. IEEE TPAMI, 2016.
[55] Gyungin Shin, Samuel Albanie, and Weidi Xie. Unsupervised salient object detection with spectral cluster voting. In CVPRW, 2022.
[56] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC, 2021.
[57] Oriane Siméoni, Chloé Sekkat, Gilles Pay, Antonin Vobecky, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. arXiv preprint arXiv:2212.07834 2022.
[58] Konstantin Sofiiuk, nya A Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3141-3145. IEEE, 2022.
[59] Sjoerd Van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-325, 2020.
[60] Vladimir Vezhnevets and Vadim Konouchine. Growcut: Interactive multi-label nd image segmentation by cellular automata. In proc. of Gruphicon, volume 1, pages 150-156. Citeseer, 2005.
[61] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large-scale generative models. 2021.
[62] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng. Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In CVPR, 2017.
[63] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng. Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 136-145, 2017.
[64] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M. Alvarez. Freesolo: Learning to segment objects without annotations. In CVPR, 2022.
[65] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. arXiv preprint arXiv:2301. 11200, 2023.
[66] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufrey- daz. Self-supervised transformers for unsupervised object discovery using normalized cut. In CVPR, 2022.
[67] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 1485-3449) IEEE, 2010.
[68] Haozhe Xie, Hongxun Yao, Shangehen Zhou, Shengping Zhang, and Wenxiu Sun. Efficient regional memory network for video object segmentation. arXiv preprint arXiv:2103.12934, 2021.
[69] Ning Xu, Brian Price. Scott Cohen, Jimei Yang, and Thomas S Huang. Deep interactive object selection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 373-381, 2016.
[70] Jianwei Yang. Anitha Kannan, Dhruv Batra, and Devi Parikh. Lr-gan: Layered recursive generative adversarial networks for image generation. arXiv preprint N.Xivv1703.01660 2017.
[71] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 879-888, 2019.
13

---

[72] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by integration. In European Conference on Computer Vision, pages 332-
348. Springer, 2020.
[73] Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[74] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315, 2018.
[75] Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
(76) Wangho Zhao, Kai Wang, Xiangxiang Chu, Fuzhao Xue, Xinchao Wang, and Yang You. Modeling motion with multi-modal features for text-based video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11737-11746. 2022.
[77] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong. Meng Wang. and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision, 2022.
[78] Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang. Deyu Meng, and Yefeng Zheng. Interactive segmentation gaussian process classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrnn Recognition, pages 5297-5306, 2019.
[79] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxi- ang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. arxiv preprint arXiv:2111.13792, 2021.
14

---

A
More Comparison with Mask-RCNN
No Object
Mask RCNN
PaintSeg
Mask RCNN
PaintSeg
Figure A: Comparison with Mask RCNN with objects beyond 80 COCO categories.
We present more results compared with supervised Mask RCNN [24]. As shown in Fig. A we compare box-prompted segmentation with Mask RCNN on objects beyond 80 COCO categories. In the shown examples, we observe that Mask RCNN has difficulty segmenting the correct shape of the object. Instead, PaintSeg provides more accurate object segmentation. As Mask RCNN is only trained on 80 COCO object categories, there is still substantial gap between the seen and the unseen. In contrast, PaintSeg solution that does not require training, which makes it more general and capable of handling new object categories.
B More Ablation Experiments
In this section, we provide additional ablation studies to illustrate the design choices of PaintSeg.
2
3
4
5
6
loU
78.8
79.2
79.6
80.1
80.6
80.8
Table A: Ablation study on the painted image number N for each step.
B.1 Sampling Number for Each Step
We average N painted images in each step to obtain the final mask prediction due the randomness of the generative painting model. We present an ablation study to illustrate the impact of the number of painted images in each step. As shown in Table A we report the performance on the ECSSD (54) dataset with coarse mask prompt from TokenCut [66]. We notice that the performance gradually improved with more painted images averaged in each step. As there is no significant difference in performance between five or six painted images used, we set the number of painted images to five in the PaintSeg process.
DINO 9 VIT-S/8
DINO- V2 466 VIS-S/14
80.6
80.0
Table B: Ablation study on image projector & used in AMCP.
B.2 Image Projector
We conduct an ablation study image projector E as illustrated in Table B We compare the widely used DINO 9 VIT-S/8 and the latest DINO (46) VIS-S/14. The results demonstrate that DINO with small patch size achieves better performance. It follows that we consider smaller patch size since PaintSeg requires fine-grained visual information. A larger patch size will blur the object boundary, resulting in performance drop.
15

---

C
More Potential Application
In this section, discuss potential applications of PaintSeg beyond prompt-guided object segmentation.
Segm.
Segm.
Difference
Difference
Original Image
Image
Remove
Remove 2
Reassembled Image
Figure B: Potential application in image edition and amodal segmentation. PaintSeg can step-by- step remove objects in the image by using the painted image I-step. With the segmented object and painted image without objects, we can freely assemble them into new image. Further, PaintSeg supports amodal segmentation, with the painting capability enabling segmentation of the occluded areas.
C.1 Image Edition
In the I-step of AMCP, the painted image will remove the target object while keeping all other contents in the image. In this way, with the segmented objects and an image without target objects, we can reassemble them into a new image as shown in Fig. B
C.2 Amodal Segmentation
As shown in Fig, B PaintSeg can layer-by-layer segment objects. By using the painted image in
I-step as the input to the next iteration, PaintSeg can attach the amodal capability. We notice that the bench is occluded by the men in Fig. B With the PaintSeg, the full shape of the bench can be segmented.
D
More Discussion about PaintSeg
D.1 Point Prompt
As depicted in Fig the object mask be derived by contrasting in/outpainted images with the original ones. In the I-step, there is a substantial difference (Fig 2 (a)) in the object area. In contrast, in the O-step, the object area has a small difference (Fig 2 (b)).
To incorporate the point information, we introduce prompt term T prompp in the contrastive potential to enhance the difference based the point location. Specifically, & Prrompeli. [i. = (xr fi (Equ
2) represents gaussian offset. Here, (z, 3) refers to the point prompt, serving as the mean, while to of the bounding box's 's height and width determine the variance (Line 159). During the I-step, we add & prompt to the contrastive potential to amplify the difference the prompted area. Conversely, during the O-step, we subtract T prompt to reduce the difference in the prompted area. This way, when we binary classify the contrastive potential (Equ 3), the point-prompted area will be classified as part of the object area.
D.2 Adversarial Mask Updating
As discussed in Sec 4.1 and Fig 2, the I-step is capable of handling false positives (FP) and the O-step for false negatives (FN). no mask updating constraints are applied, as the mask updating may not be accurate in the first several steps, new FP can be introduced in I-step. However, the new EP cannot
16

---

be addressed in the following O-step and has to wait for the next I-step to be handled. In contrast, if we constrain the I-step only shrinks the object mask, no FP will be introduced in the I-step and the remaining FN can be addressed in the following O-step. Similarly, we constrain O-step only to expand the object mask thus no FN will be introduced. We refer to the "shrink" I-step and "expand" in O-step as adversarial mask updating. By employing the adversarial mask updating, we can address errors (FP/FN) that occurred during each step immediately without degradation.
D.3
In PaintSeg, introduce a latent variable point which is characterized by off-the-shelf generative model p(4 o MM conditioned on an image I and a mask M. o represents Hadamard product. In our method, we leverage the AMCP process to estimate and convert the latent variable 1ponnt into mask prediction AM with alternating I-step and O-step. Mathematically, both I-step and O-step can be formulated as expectation- expectaioo-maxxmiizaion.
process.
Expectation: We introduce latent variable paint in the proposed PaintSeg which is modeled by an off-the-shelf generative painting model pl pl.pasne I o M). We assume the generative model will pick the most likely outcome [paint given I and M for every step.
Maximization: After obtaining the latent variable painne. define contrastive potential & and utilize clustering to binarize the mask. Mathematically, the contrasting and clustering TMTlo (Ipavst.II.M)
processes maximize posteriori probability p(M I paints I)
Although term I-step and O-step separately, they can be formulated the same EM process. ground PaintSeg advances the predicted mask to the ground truth by iteratively conducting the EM process in each step.
D.4 Contrastive Potential
There are three terms in the contrastive potential 4 paintt Peator and Q prompti
T paiint As analyzed in Sec 4.1, we can segment the object by contrasting the original and in/outpainted images. In this way, we formulate 4 paint to describe the difference between the original and in/outpainted images and maximize
locate the object.
4 Pcotor: As an image projector, i.e.. DINO, leveraged, the object segmentation from solely o "pint is achieved at patch level. Therefore, we further introduce a color potential & cotor to incorporate pixel-level color information better segment the object boundary. Pcolor describes the probability of a pixel belonging to the foreground or background region based on the color similarity. By combining T paint and t Peolor, we can achieve high-quality object segmentation with AMCP.
d prompe:: In addition, the & prompt is introduced to provide additional location information to guide the PaintSeg locate the target object. We will add the discussion in the revision.
E Difference with Previous Segmentation Approaches
In this section, we discuss the major differences between the proposed PaintSeg and previous object segmentation methods as follows.
Discriminative v.s. Generative + Discriminative. Conventional object segmentation is a dis- criminative task that leverages a neural network 0 to model the conditional probability of the object mask . given the image I as condition Po (a(M [2). In PaintSeg, we have mask, paint, and contrast operations in each step. Specifically, in paint operation, we enroll a generative model to estimate painted image Ipoint with mask M and image I conditions. After that, the mask can be obtained by comparing the generated image with the original one with a contrastive potential &. As discussed in Section D.3] the paint operation is generative process to estimate latent variable p( p(Tpaine| |T M) and the contrast operation is a discriminative process to obtain a mask prediction based on P(M) 7). PaintSeg achieves training-free by constructing a bridge to generative painting models which permits object shape consistency and background content consistency.
17

---

Pixel v.s. Pixel difference. Conventional object segmentation leverages network to project an image the feature space and then binarize (cluster) each pixel into foreground background classes. Differently, instead of directly clustering over the input image, PaintSeg utilizes the difference between the painted and original image, as proxy, to leverage the object shape prior and background consistency. The contrastive scheme is rooted in the decomposable nature of images and paves way to incorporate generated images to segment objects.
Training v.S. Training-free. Conventional object segmentation approaches train the neural network to segment objects requiring time-consuming and expensive data labeling. Some unsupervised segmentation methods 4 IO. 61] find segment from generative model while they typically require training network on top of the generative model. Instead, our method a training ffeee unsupervised method that learns to segment objects from generative painting model. We consider the PaintSeg provides a way to bridge the generative model and segmentation which may inspire future research.
F Failure Case Analysis
We analyze the failure case here. As shown in Fig. C we visualize fail- ure case when using point as the prompt. We notice the adjacent car segmented as false positive, which due to the semantic and visual simi-
larity between the target and false pos- itive cars. Despite our method is capa-
Figure C: Illustration of failure case.
ble of handling multiple objects with point prompt (right of Fig. C crowded scenarios can make it
difficult to segment the accurate object boundary. However, the issue can be overcome through box prompt.
G More Visualization
In this section, we demonstrate more visualization of PaintSeg. We show more qualitative results with box prompt in Fig. D with point prompt in Fig. Big.Bern and with coarse mask prompt in Figs. F and G
18

---

STOP
CENTRETHNS TNN OFORGE TIAN
port-
Figure D: More visualization of PaintSeg with box prompt COCO MVal.
19

---

Figure E: More visualization of PaintSeg with point prompt. The point prompt is illustrated by the red point on the image on DAVIS and Berkeley and GrabCut.
20

---

Prompt
Mask
Prompt
Mask
YORK BUILDINGS WC2
YORK BUILDINGS WC2 MECORIITT
40
40
NEXT
NEXT
Figure F: More visualization of PaintSeg with coarse mask prompt on ECSSD.
21

---

Prompt
Mask
Prompt
Mask
SLOW
SLOW
1 UR
1 UR
SOUTH
SOUTH
1
1
DESPACIO
DESPACIO
MURSERY RAPPED AFTER CHILDREN ESCAPE
BURSERY RAPPED AFTER CHILDREN ESCAPE
Figure G: More visualization of PaintSeg with coarse mask prompt ECSSD.
22


Reviews:
[{"review_id": "yox29O9MYG", "reviewer": "Reviewer_Xo6H", "summary": "This paper designs a novel unsupervised segmentation method (AMCP) without training via image painting. Its main process is to mask the image based on the specified prompt, recover the image using a generative model, and determine the foreground mask and background mask of the image by comparing the original and inpainted images. Specifically, based on the specified prompt, the image is first masked and then recover using a generative model. Finally, by comparing the original and inpainted images, the foreground mask and background mask of the image are determined. Specifically, it designed I-step and O-step respectively. In the I-step, the background of the image is restored from the deleted object region, and the foreground mask can be obtained by comparing it with the original image. Conversely, in the O-step, the background mask can be obtained by drawing the foreground in the image. Finally, a large number of experiments fully demonstrate the effectiveness of the method.", "questions": "shown in weaknesses", "limitations": "shown in weaknesses", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of obtaining the segmentation result of an image by comparing the difference between the original and inpainted images is interesting. \n2. The manuscript is well organized and has several experiments.", "weaknesses": "1. Mt+ and Mt− serve as the dilation and erosion masks of Mt, and are important conditions for updating the regions near the foreground-background boundary. The description in the implementation details cannot show the process and loses the discussion of their necessity. \n2. Since AMCP first performs the I-step and the input init mask needs to include the false positives region, it is not possible to directly perform the I-step for point or scribble prompts. In this case, is it feasible to swap the order of the I-step and O-step?\n3. DINO and Fully Connected CRF are used in Line 150 and 154 respectively, and they play a very important role in AMCP. What would be the impact on AMCP if they were replaced? DINO's features are highly robust, and replacing DINO with other models can be used to verify if it is DINO that has improved the upper limit of AMCP.\n4. The core of this paper is to segment images based on a specific prompt, which is the same setting as SAM and SEEM. Therefore, fair comparisons should be made with them in comparative experiments. Currently, the comparison method introduced in the paper is not fair, as PaintSeg requires a specific prompt to be given, while most of the unsupervised methods being compared do not have this setting.\n5. Although AMCP is a training-free method, it involves many hyperparameters and other introduced models. It is recommended to add pseudocode to more clearly demonstrate the detailed process of AMCP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. The role and necessity of updating near the foreground-background boundary (adversarial mask updating).**\n\nWe would like to further clarify the necessity of adversarial mask updating. As discussed in Sec 4.1 and depicted in Fig 2, the I-step effectively addresses false positives (FP) and the O-step for false negatives (FN). \n\n- *Without mask updating constraints*: Since mask updating may lack accuracy in the initial steps, new FPs can emerge after the I-step. However, these new FPs cannot be rectified in the subsequent O-step and must await the next I-step for resolution. Similarly, new FNs can emerge during the O-step.\n\n- *With mask updating constraints*: By constraining the I-step solely to \"shrink\" the object mask, new FPs will not emerge in the I-step. The remaining FNs can then be addressed in the following O-step. Similarly, by limiting the O-step to \"expand\" the object mask, new FNs will not be introduced.\n\nWe refer to the \"shrink\" in I-step and \"expand\" in O-step as adversarial mask updating. By employing the adversarial mask updating, we can promptly address errors (FP/FN) that arise during each step without degradation.\n\nThe terms \"dilation\" and \"erosion\" in Section 5.2 (implementation details) pertain not to adversarial mask updating, but to the morphological operations employed for filtering sparse points after binarizing the contrastive potential into the object mask. We will revise the wording in the revised version.\"\n\n---\n\n**2. Since AMCP first performs the I-step and the input init mask needs to include the false positives region, it is not possible to directly perform the I-step for point or scribble prompts. Is it feasible to swap the order of I-step and O-step for point and scribble prompts?**\n\nUpon initially performing the O-step using point and scribble prompts, we observed that the quality of the outpainted image lacks robustness due to the limited conditioned area within the point/scribble regions. To address this, we establish the entire image area as the initial mask for point and scribble prompts (Lines 233-235) and initiate AMCP from the I-step. Object localization is guided by the prompt term $\\Phi_{prompt}$ in the contrastive potential $\\Phi$. The process of point-prompted AMCP is visualized in Fig A (found in the PDF file within the general response).\n\nWe recognize that Fig 2 might be misleading since it illustrates the rationale behind contrastive painting using examples of box and scribble prompts. In the revision, we will replace the terms \"box\" and \"scribble\" in Figure 2 with \"mask with false positives\" and \"mask with false negatives\".\n\n---\n\n**3. The impact on AMCP if DINO and CRF are replaced.**\n\nAMCP is based on the difference between the contents in the original and in/outpainted images. We leverage an image projector, i.e., DINO, to capture object semantic information to coarsely locate the object and utilize the CRF to capture pixel-level color information to better segment the object boundary.\n\nAs shown in Tab A and Tab B, we conduct ablation studies to replace DINO with MoCov2 [11] and SwAV [7] and replace CRF [30] with the bilateral solver [2] respectively. We notice that the performance of AMCP only exhibits a slight variation when these replacements are applied.\n\n|Image Projector| MoCov2 [11] | SwAV [7] | DINO [8] |\n|:----:| :----: | :----: | :----: |\n| IoU | 80.1 | 79.7 | 80.6 |\n\nTable A: Ablation on image projector with coarse mask prompts on ECSSD. \n\n|Color potential|  None  | BS [2] | CRF [30]|\n|:----:| :----: | :----: | :----: |\n| IoU | 71.4 | 79.1 | 80.6 |\n\nTable B: Ablation on the method to calculate color potential with coarse mask prompts on ECSSD. BS: bilateral solver [2].\n\n---\n\n**4. A fair comparison should be made with SAM. The comparison in the paper is not fair.**\n\nWe report the performance of SAM (ViT-Base) as shown in Tab C while we think this comparison is not exactly fair as SAM is a supervised method with (1) the large-scale supervised training with labeled data, and (2) the large number of parameters. Different from the supervised pipeline employed by SAM, PaintSeg demonstrates that satisfactory segmentation can be achieved in a fully unsupervised manner by contrasting original and in/outpainted images. We believe the findings from PaintSeg remain valuable even if its performance is suppressed by SAM.\n\nFor comparison in Tab 2 and Tab 3 in the paper, all methods are equipped with the same prompts (point or box) to ensure a fair comparison. Specifically, all compared point-prompt methods are designed to accept point prompts (following the conventional evaluation protocol). For box-prompt methods, we augment the compared methods by first cropping the object box region and conducting segmentation on the cropped images. In this way, we consider that those comparisons are fair and can reflect the effectiveness of PaintSeg.\n\n| Method | GrabCut (point) | Berkeley (point) | DAVIS (point) |  PASCAL VOC (box) | MVal (box) |\n|:----| :----: | :----: | :----: | :----: | :----: |\n| SAM [29] | 86.7 | 83.3 | 71.1 | 66.4 | 83.0 | \n| PaintSeg| 84.4 | 70.0 | 69.4 | 59.7 | 69.6 |\n\nTable C: Comparison with SAM for point and box prompted segmentation.\n\n---\n\n**5. Pseudocode for more clear demonstration.**\n\nThanks for your suggestion. We demonstrate the pseudocode of AMCP in Algo A (in the PDF attached in the general response). We will add the pseudocode in the revision. "}, {"review_id": "RLzeiGwWjl", "reviewer": "Reviewer_fhrH", "summary": "This paper presents a new unsupervised method for object segmentation, named PaintSeg. Given the initial mask, the proposed PaintSeg adopts adversarial masked contrastive painting (AMCP) to generate the foreground and background masks through the contrastive relations between the original image and the painted image. The AMCP process involves two steps: I-step for the foreground mask and O-step for the background mask. The proposed PaintSeg based on the off-the-shelf diffusion models does not rely on training with labeled samples but generate segmentation results through iterative optimization with the I-step and the O-step. Experiments show the good performance achieved by the proposed PaintSeg.", "questions": "1. Considering that the proposed PaintSeg adopts diffusion models for inpainting, I'm concerned about whether PaintSeg is sensitive to the generative models and whether different generative models affect the final quality or the convergence speed.\n2. How about the latency of generating the segmentation mask.\n3. I suggest the authors add the evaluation results of SAM with the same prompts, which can show the upper bound or the superiority of the proposed approach. ", "limitations": "The authors have clarified the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using inpainting and outpainting for foreground and background masks is interesting and novel to me.\n2. This paper addresses unsupervised image segmentation in a training-free manner with pre-trained generative models.\n3. This paper presents the optimization steps with contrastive potential and adversarial steps.\n4. The proposed approach PaintSeg is effective and achieves good results on several benchmarks.\n", "weaknesses": "1. Firstly, it's a nice work and there is no significant weakness.\n2. Based on LDM, the proposed PaintSeg takes much time to iteratively generate masks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. Whether PaintSeg is sensitive to the generative models?**\n\nWe consider PaintSeg is not sensitive to generative models. PaintSeg segments objects by contrasting original and in/outpainted images. Even if different generative models may tend to generate objects in different styles, as long as the generated object follows the realistic distribution, PaintSeg can decode it with the EM-like process. As shown in Tab A, we conduct an ablation study to replace different generative models in AMCP. We notice that, for the generative model with similar performance (Stable diffusion1.5, SD2.1, Kandinsky 2.1), AMCP is robust with fixed hyperparameters. However, for the diffusion model with obviously inferior performance (latent diffusion), AMCP performance will degrade accordingly. \n\n|Generative model| Box |Mask |\n|:----| :----: | :----: |\n| Latent diffusion |  68.4 | 76.8 |\n| Kandinsky 2.1| 71.5 | 79.4 |\n| Stable diffusion 2.1 |  72.2 | 80.1 |\n| Stable diffusion 1.5 |  71.0 | 80.6 |\n\nTable A: Ablation of generative models on ECSSD dataset.\n\n---\n\n**2. Latency of generating the segmentation mask?**\n\nWe report the computing time of AMCP tested on NVIDIA Tesla V100 32G GPU as shown in Tab B. We notice the most time-consuming part is the multi-step diffusion inference. As AMCP is not sensitive to generative models, with the extensive research in diffusion sampling [R1,R2], we consider the speed of AMCP can be further improved with more advanced diffusion models in the future.\n\n| In/Outpainting (diffusion) | Calculating $\\Phi$ (CRF) | K-means | Total |\n|:----:| :----: | :----: | :----: |\n| 13.91s | 0.46s | 0.02s | 14.39s |\n\nTable B: Latency of components for each iteration of AMCP.\n\n[R1] Consistency Models, ICML 2023\n\n[R2] One-Step Diffusion Distillation via Deep Equilibrium Models, ICML 2023 Workshop\n\n---\n\n**3. Comparison with SAM.**\n\nWe report the performance of SAM with the same prompts as used in PaintSeg in Tab C. We notice that SAM shows superior performance compared to PaintSeg which can be attributed to two main factors: (1) the large-scale supervised training with labeled data, and (2) the large number of parameters. Different from the supervised pipeline employed by SAM, PaintSeg demonstrates that satisfactory segmentation can be achieved in a fully unsupervised manner by contrasting original and in/outpainted images. We consider the findings of PaintSeg to remain valuable even if its performance is suppressed by SAM.\n\n| Method | GrabCut (point) | Berkeley (point) | DAVIS (point) |  PASCAL VOC (box) | MVal (box) |\n|:----| :----: | :----: | :----: | :----: | :----: |\n| SAM [29] | 86.7 | 83.3 | 71.1 | 66.4 | 83.0 | \n| PaintSeg| 84.4 | 70.0 | 69.4 | 59.7 | 69.6 |\n\nTable C: Comparison with SAM for point and box prompted segmentation."}, {"review_id": "9Nwchlu6bs", "reviewer": "Reviewer_9reF", "summary": "The following work proposes an iterative approach to converting any off-the-shelf inpainting/outpainting capable models to perform segmentation. The proposed method alternates between two approaches to separating the segmentable foreground object from the background. In the I(npainting) step, they mask out the foreground object and inpaint the gap, then locally contrast the in-painted image against the original. The inpainting will fill the hole with an extension of the background instead of the masked out foreground object, thus the disparity between the two images gives rise to a segmentation mask for said object. The O(utpainting) step is similar but expands outwards from a sample of the foreground object. The disparity between the outpainted image and the original image will be large on the non-foreground pixels, resulting in an approximate inverted result from the I-step.  Experiments suggest significant improvements over recent methods with limited to no additional training.", "questions": "Address weakness section", "limitations": "Adequately addressed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Approach appears principled and builds upon older methods involving conditional random fields\n- Strong results with minimal (or none at all?) additional supervised training necessary; potential for strong generalization", "weaknesses": "To me, the primary concerns of this work lie in the clarity of the writing:\n- While I understand why the authors chose to refer to the approach as adversarial masking, I don't think this is correct. The inpainting and outpainting steps come from opposite direction but have a more complimentary relationship than an adversarial one. The formulation in (6) similarly does not exhibit signs of any adversarial interaction between opposing objectives, but would be more aptly described an alternating minimization scheme of two separate constraint terms. \n- Overall, I felt that the writing could have been more clear. In section 2 on contrastive potentials, the authors assign the sum of potential functions to the term contrastive potentials. Critically missing is how the potential values relate to the segmentation task itself, presumably through an argmax obejctive. In addition, while the terms inpainting and outpainting are generally accepted terminology, I don't believe it is common to refer to the resulting image as the \"painted image\", which definitely led to some reading confusion on my end.\n- It is also unclear to me where the cluster centers \\mu_k come from in the k-means mask update step in section 4.3\n- Some additional missing ablations include ablations over the lambda weights for individual potential functions, as well as choice of generative model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. Does PaintSeg require supervision?**\n\nIn the entire pipeline, PaintSeg does not require any supervision. We find it intriguing to demonstrate that satisfactory segmentation can be achieved in a fully unsupervised manner by contrasting original and inpainted/outpainted images\n\n---\n\n**2. The term \"adversarial masking\" is not appropriate as the relation between inpainting and outpainting is more complementary than adversarial.**\n\nWe would like to clarify that the \"adversarial mask updating\" does not refer to the alternating I- and O-steps but the mask updating constraints (Equ 4: only shrink mask in I-step and expand mask in O-step). As discussed in Sec 4.1 and Fig 2, the I-step is capable of handling false positives (FP) and the O-step for false negatives (FN). \n\n- *Without mask updating constraints*: Since mask updating may lack accuracy in the initial steps, new FPs can emerge after the I-step. However, these new FPs cannot be rectified in the subsequent O-step and must await the next I-step for resolution. Similarly, new FNs can emerge during the O-step.\n\n- *With mask updating constraints*: By constraining the I-step solely to \"shrink\" the object mask, new FPs will not emerge in the I-step. The remaining FNs can then be addressed in the following O-step. Similarly, by limiting the O-step to \"expand\" the object mask, new FNs will not be introduced.\n\nWe refer to the \"shrink\" in I-step and \"expand\" in O-step as adversarial mask updating. By employing the adversarial mask updating, we can promptly address errors (FP/FN) that arise during each step without degradation. We will further clarify the term \"adversarial\" in the revision.\n\n---\n\n**3. How do contrastive potential values relate to the segmentation itself? The term \"painted image\" is confusing.**\n\nThere are three terms in the contrastive potential $\\Phi_{paint}$, $\\Phi_{color}$ and $\\Phi_{prompt}$.\n\n- $\\Phi_{paint}$: As analyzed in Sec 4.1, we can segment the object by contrasting the original and in/outpainted images. In this way, we formulate $\\Phi_{paint}$ to describe the difference between the original and in/outpainted images and maximize it to locate the object. \n\n- $\\Phi_{color}$: As an image projector, i.e., DINO, is leveraged, the object segmentation from $\\Phi_{paint}$ is achieved at a patch level. Therefore, we further introduce a color potential $\\Phi_{color}$ to incorporate pixel-level color information to better segment the object boundary. $\\Phi_{color}$ describes the probability of a pixel belonging to the foreground or background region based on the color similarity. By combining $\\Phi_{paint}$ and $\\Phi_{color}$, we can achieve high-quality object segmentation with AMCP. \n\n- $\\Phi_{prompt}$: In addition, the $\\Phi_{prompt}$ is introduced to provide additional location information to guide the PaintSeg to the target object. \n\nThank you for pointing out the confusion about the \"painted image\". We will revise the term as \"in/outpainted image\" in the revision to enhance the readability.\n\n---\n\n**4. Where does cluster center $\\mu_k$ come from k-means?**\n\nSince we conduct k-means on the contrastive potential values $\\Phi[h,w]\\in\\mathbb{R}^{1}$, the cluster center is the average value of all the samples in a cluster. For the 2-clusters case, the $\\mu_k$ refers to the average value of $\\Phi$ in the foreground or background regions. We will revise $\\mu_k$ as the average value of all the samples in the $k$- cluster in our revision.\n\n---\n\n**5. Ablation on lambda weights for individual potential functions and the choice of the generative model.**\n\nWe conduct the ablation study to investigate the influence of potential function as shown in Tab A. Additionally, we ablate the weights to combine $\\Phi_{paint}$ and $\\Phi_{color}$ as shown in Tab B. We notice each term in the contrastive potential benefits the performance.\n\nAs shown in Tab C, we ablate on different generative models. For the generative model with similar performance (SD1.5, SD2.1, Kandinsky 2.1), we notice that the performance of PaintSeg is robust. However, if the diffusion model has obviously inferior performance (latent diffusion), AMCP performance will degrade accordingly.\n\n| $\\Phi_{paint}$ | $\\Phi_{color}$ | $\\Phi_{prompt}$ | Mask | Box | Point |\n| ----| :----: | :----: | :----: | :----: | :----: | \n| &#10004; |                 |                 | 71.4 | 67.8 | - |\n| &#10004; | &#10004; |                 | 80.6 | 70.2 | - |\n| &#10004; |                 | &#10004; | -      | 68.1 | 56.9 |\n| &#10004; | &#10004; | &#10004; | -      | 71.0 | 60.8 |\n\nTable A: Ablation of the components in the contrastive potential. For the mask prompt, $\\Phi_{prompt}$ is not used (Line 156-157) and, for the point prompt, $\\Phi_{prompt}$ is necessary to locate the object.\n\n| $\\lambda_{paint}$ | $\\lambda_{color}$ | Mask |\n| ----| :----: | :----: | \n| 0.80 | 0.10 | 79.7 |\n| 0.80 | 0.15 | 80.1 |\n| 0.80 | 0.20 | 80.6 |\n| 0.80 | 0.25 | 80.4 |\n\nTable B: Ablation of the weights of contrastive potential on ECSSD.\n\n|Generative model| Mask | Box |\n|:----| :----: | :----: |\n| Latent diffusion | 76.8 | 68.4 |\n| Kandinsky 2.1| 79.4 | 71.5 |\n| Stable diffusion 2.1 | 80.1 | 72.2 |\n| Stable diffusion 1.5 | 80.6 | 71.0 |\n\nTable C: Ablation of generative models on ECSSD dataset. "}, {"review_id": "Mi694Ztotx", "reviewer": "Reviewer_CwUQ", "summary": "The author proposes a training-free segmentation technique AMCP, which leverages off-the- shell diffusion model to refine the corase segmentation prompt iteratively. AMCP contains three main components: Inpainting step, outpainting step and adversarial updating. The paper gives detail ablation of AMCP and demonstrate its effectiveness on sevaral benchmarks.", "questions": "1. The contrastive potential consists of three items. How does each item affect the segmentation result？", "limitations": "Authors address the limitations of their work and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper achieves a successful integration of an existing denoising diffusion model, exploiting its potential in segmentation tasks by inpainting and outpainting techniques, resulting in promising performance.\n2. This paper conducts comprehensive comparisons with other segmentation approaches and provides a detailed ablation analysis of the techniques employed in AMCP", "weaknesses": "1. The main contribution to the segmentation ability stems from the diffusion model. However, the low speed of the sampling process in the diffusion model poses a significant obstacle for real-time applications. As AMCP lacks classification and object discovery capabilities, it is better suited as a labeling tool or an interactive segmenter, the latter of which demands even higher real-time performance. However, achieving the performance demonstrated in the paper requires AMCP to undergo several diffusion steps (5), further diminishing its effectiveness.\n\n2. I notice that AMCP is sensitive to the cluster center number, which leads to about 20 mIoU variations. In the paper, AMCP is fixed to use the pre-training parameters of one specific diffusion model. When employing different diffusion models, the impact of each inpainting varies. Whether the hyperparameters of AMCP reasoning also need to be adjusted accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. The segmentation ability stems from the diffusion model while the sampling speed of the diffusion model obstacles for real-time applications.**\n\nWe acknowledge that PaintSeg is not efficient. While PaintSeg actually relies on high-quality in/outpainted images, without necessitating a specific generative model. Considering the swift advancements in diffusion sampling [R1, R2], we anticipate that the speed of AMCP can be further enhanced in the future by leveraging more advanced diffusion models.\n\nIn addition to the value of applications, PaintSeg also reveals that satisfactory segmentation can be achieved in a fully unsupervised manner by contrasting original and in/outpainted images which can be of interest to the community.\n\n[R1] Consistency Models, ICML 2023\n\n[R2] One-Step Diffusion Distillation via Deep Equilibrium Models, ICML 2023 Workshop\n\n---\n\n**2. Performance is sensitive to cluster center numbers. When employing different diffusion models, whether the hyperparameters of AMCP reasoning also need to be adjusted accordingly?**\n\nSince only 2 categories - foreground and background are required to discriminate, a cluster number of 2 is the most reasonable choice. We ablate on the cluster number to verify if an ambiguous region between foreground and background exists when calculating contrastive potential $\\Phi$. As AMCP with a cluster number larger than 2 will discard an ambiguous region in each step, the performance will largely drop if the ambiguous region does not exist (we verify that a cluster number of 2 leads to the best result). \n\nWe consider that the core of AMCP is based on contrasting in/outpainted and original images and the hyperparameters of AMCP do not have a model-specific design. In this way, if the generated image quality of the diffusion model does not have a big gap, no further hyperparameter adjustment is required. As shown in Tab A, we ablate on different generative models. For the generative model with similar performance (SD1.5, SD2.1, Kandinsky 2.1), we notice that AMCP is robust with fixed hyperparameters. However, for the diffusion model with obviously inferior performance (latent diffusion), AMCP performance will degrade accordingly.\n\n|Generative model| Mask | Box |\n|:----| :----: | :----: |\n| Latent diffusion | 76.8 | 68.4 |\n| Kandinsky 2.1| 79.4 | 71.5 |\n| Stable diffusion 2.1 | 80.1 | 72.2 |\n| Stable diffusion 1.5 | 80.6 | 71.0 |\n\nTable A: Ablation of generative models on ECSSD dataset.\n\n---\n\n**3. How does each item in contrastive potential affect the segmentation result?**\n\nThere are three terms in the contrastive potential $\\Phi_{paint}$, $\\Phi_{color}$ and $\\Phi_{prompt}$.\n\n- $\\Phi_{paint}$: As analyzed in Sec 4.1, we can segment the object by contrasting the original and in/outpainted images. In this way, we formulate $\\Phi_{paint}$ to describe the difference between the original and in/outpainted images and maximize it to locate the object. \n\n- $\\Phi_{color}$: As an image projector, i.e., DINO, is leveraged, the object segmentation from $\\Phi_{paint}$ is achieved at a patch level. Therefore, we further introduce a color potential $\\Phi_{color}$ to incorporate pixel-level color information to better segment the object boundary. $\\Phi_{color}$ describes the probability of a pixel belonging to the foreground or background region based on the color similarity. By combining $\\Phi_{paint}$ and $\\Phi_{color}$, we can achieve high-quality object segmentation with AMCP. \n\n- $\\Phi_{prompt}$: In addition, the $\\Phi_{prompt}$ is introduced to provide additional location information to guide the PaintSeg to the target object. \n\nWe conduct an ablation study on components in contrastive potential as shown in Tab B. Compared to solely leverage $\\Phi_{paint}$, combining $\\Phi_{paint}$ and $\\Phi_{color}$ yields better performance by refining the patch-level mask to pixel-level. For $\\Phi_{prompt}$, it provides additional location information thus benefits the performance.\n\n| $\\Phi_{paint}$ | $\\Phi_{color}$ | $\\Phi_{prompt}$ | Mask | Box | Point |\n| ----| :----: | :----: | :----: | :----: | :----: | \n| &#10004; |                 |                 | 71.4 | 67.8 | - |\n| &#10004; | &#10004; |                 | 80.6 | 70.2 | - |\n| &#10004; |                 | &#10004; | -      | 68.1 | 56.9 |\n| &#10004; | &#10004; | &#10004; | -      | 71.0 | 60.8 |\n\nTable B: Ablation of the components in the contrastive potential. For the mask prompt, $\\Phi_{prompt}$ is not used (Line 156-157) and, for the point prompt, $\\Phi_{prompt}$ is necessary to locate the object (Line 234-235)."}, {"review_id": "tEyd3B25fG", "reviewer": "Reviewer_1n6h", "summary": "This paper proposed to interactively use inpainting (I-step) and outpainting (O-step) to achieve foreground masks without training.\nFormally, the I-step refines the false positive masks while the O-step tackles false negative ones. The authors proposed Contrastive Potential to evaluate the differences between the origin and inpainted images, and gradually refine the segmentation.\nVarious prompts are supported to start the I-step and O-step (point, box, scribble).\nExperiment results show the effectiveness of the proposed method.", "questions": "This paper enjoys good novelty. However, some presentations ($\\mu_k$, point prompt) and experiments (efficiency, Contrastive Potential's ablation) are clear enough. I would increase my score if the authors could reasonably clarify them.", "limitations": "Yes the authors have discussed about them", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Leveraging inpainting and outpainting to address the segmentation is novel and interesting.\n2. Quantitative results show the effectiveness of the proposed method. It enjoys good improvements for untrained algorithms.\n3. The mathematical formulation of EM is interesting.", "weaknesses": "1. I think this method is not efficient because it used a diffusion-based method to inpaint/outpaint iteratively. But the authors did not provide a comparison of efficiency.\n2. Though the authors claim that their method could work with prompts: point, box, and scribble, they did not provide sufficient illustration about the usage of point prompts in Fig.2 and Sec4. In my opinion, both inpainting and outpainting fail to achieve proper results based on only one point prompt. \n3. There are not enough ablations to confirm the effectiveness of Contrastive Potential (paint, color, prompt). It is unclear how to formulate Eq. (2) in the absence of point prompts.\n4. Eq.(3) is a bit difficult to understand. It seems like $\\mu$ is supposed to represent the center position of a cluster, but I'm not sure what the notation \"$argmax(\\mu_k)$\" means. Personally, I believe it would be more helpful if the authors presented the average value of all the samples in a cluster as the value for $\\mu,\" instead of just the cluster center.\n5. A more equitable experiment would be to classify the competitors in Tab.2 into those with prompts and those without. It would be unjust to compare the proposed method with competitors lacking prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. The method is not efficient. Lack of comparison of efficiency.**\n\nWe acknowledge that PaintSeg is not efficient. We report the computing time for each component of AMCP tested with NVIDIA Tesla V100 32G GPU in Tab A. We notice that the majority of time is consumed by the diffusion model. \n\nAs shown in Tab B, we ablate on the impact of different generative models on PaintSeg. For the generative model with similar performance (SD1.5, SD2.1, Kandinsky 2.1), we notice that the performance of PaintSeg remains robust. This suggests that PaintSeg holds promise for facile integration with various generative models in a plug-and-play manner. Considering the swift advancements in diffusion sampling [R1,R2], we anticipate that the speed of PaintSeg can be further enhanced in the future by leveraging more advanced diffusion models. \n\n| In/Outpainting (diffusion) | Calculating $\\Phi$ (CRF) | K-means | Total |\n|:----:| :----: | :----: | :----: |\n| 13.91s | 0.46s | 0.02s | 14.39s |\n\nTable A: Latency of components for each iteration of AMCP.\n\n|Generative model| Mask | Box |\n|:----| :----: | :----: |\n| Latent diffusion | 76.8 | 68.4 |\n| Kandinsky 2.1| 79.4 | 71.5 |\n| Stable diffusion 2.1 | 80.1 | 72.2 |\n| Stable diffusion 1.5 | 80.6 | 71.0 |\n\nTable B: Ablation of generative models on ECSSD dataset.\n\n[R1] Consistency Models, ICML 2023\n\n[R2] One-Step Diffusion Distillation via Deep Equilibrium Models, ICML 2023 Workshop\n\n---\n\n**2. Not sufficient illustration about the usage of point prompts. In my opinion, both inpainting and outpainting fail to achieve proper results based on only one point.**\n\nSorry for the confusion. We would like to clarify that, the entire image area is set to be the init mask for point and scribble prompts (Line 233-235), and the prompt term $\\Phi_{prompt}$ in contrastive potential $\\Phi$ serves as the guidance to locate the object. In this way, we can avoid outpainting with a small conditioned area. We visualize the process of point-prompted AMCP in Fig A (in the PDF file within the general response). \n\nWe notice that Fig 2 might be misleading as we illustrate the rationale of contrastive painting with examples of box and scribble prompts. We will replace the \"box\" and \"scribble\" in Fig 2 with \"mask with false positives\" and \"mask with false negatives\" in the revision.\n\n---\n\n**3. Ablation of contrastive potential. How to formulate it with point prompts?**\n\nWe conduct an ablation study on contrastive potential as shown in Tab C. Compared to solely leverage $\\Phi_{paint}$, combining $\\Phi_{paint}$ and $\\Phi_{color}$ yields better performance by refining the patch-level mask to pixel-level. For $\\Phi_{prompt}$, it provides additional location information thus benefits the performance. By initializing the entire image as the input mask for point and scribble prompts, the contrastive potential for them can be calculated in the same way as box and mask prompts.\n\n| $\\Phi_{paint}$ | $\\Phi_{color}$ | $\\Phi_{prompt}$ | Mask | Box | Point |\n| ----| :----: | :----: | :----: | :----: | :----: | \n| &#10004; |                 |                 | 71.4 | 67.8 | - |\n| &#10004; | &#10004; |                 | 80.6 | 70.2 | - |\n| &#10004; |                 | &#10004; | -      | 68.1 | 56.9 |\n| &#10004; | &#10004; | &#10004; | -      | 71.0 | 60.8 |\n\nTable C: Ablation of the components in the contrastive potential. For the mask prompt, $\\Phi_{prompt}$ is not used (Line 156-157) and, for the point prompt, $\\Phi_{prompt}$ is necessary to locate the object.\n\n---\n\n**4. The meaning of $\\mu$. Better to present the average value of all the samples in a cluster as the value for $\\mu$ instead of just the cluster center.**\n\nThanks for your suggestion! Yes, it would be more clear to describe $\\mu$ as the average value of all samples in a cluster. We will revise it in the revision.\n\n---\n\n**5. Better to classify competitors into those with/without prompts. Unjust to compare the proposed method with competitors lacking prompts in Tab 2.**\n\nThank you for your suggestion. We consider that all methods in Tab 2 are equipped with the same point prompt (centroid of the GT mask, following convention) including training-free methods such as GrowCut [54]. Therefore, we believe the comparison in Tab 2 is conducted fairly.\n"}]

Author rebuttal:
We sincerely thank the reviewers for their valuable time and efforts.

We include the additional figure (Fig A) and pseudo code (Algo A) in the PDF file along with additional tables in the response for each question. We hope those responses could clarify your confusion, and we are more than happy to provide further explanations if needed.

Meta-review: The work received favorable reviews. The reviewers found the idea of using alternating inpainting and outpainting steps to create segmentation masks without additional supervision novel and principled. The initial clarity concerns raised by the reviewers were well addressed by the rebuttal.

The AC agrees with the reviewers assessment - the authors propose a creative training-free approach for using generative models for segmentation. The AC requests the authors to incorporate the reviewers feedback and the rebuttal into the final version to further enhance the quality and impact of their work. The AC is pleased to recommend accepting!
Decision: Accept (poster)