SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[17] as our  $\mathcal{E}$ . We use [31] as our  $\mathcal{C}(\cdot)$  to calculate  $\Phi_{color}$ . If no specification, for all experiments, the masked contrastive painting starts from the I-step and updates for 5 steps. We set the number of cluster centers to 3 in the first three steps for point, box and scribble prompts otherwise 2. We set  $\lambda_{paint} = 0.8$ ,  $\lambda_{color} = 0.2$  and  $\lambda_{prompt} = 0.2$  if in I-step and  $\lambda_{prompt} = -0.2$  if in O-step. We average  $N = 5$  painted images to obtain the updated mask for each step. The  $\sigma_x$  and  $\sigma_y$  are set to  $\frac{1}{10}$  of the width and height of the bounding box of the current stage mask respectively.  $\Delta^{+}$ and  $\Delta^{-}$ are the neighbors 32 pixels outside and inside the object boundary. We leverage dilation and erosion to filter out sparse points for each iteration. The kernel size is set to 5. For the mask and box prompts, we set the prompt as the initial mask. For the point and scribble prompts, we set the entire image as the initial masked region. The images are padded to  $512\times 512$  to fit the generative inpainting model.
Coarse mask prompt. Since the usage of the ground-truth coarse mask as a prompt is rare, we evaluate PaintSeg on two unsupervised salient object detection benchmarks and leverage the coarse mask generated from TokenCut [66] as our prompt. As shown in Table 1, PaintSeg achieves encouraging performance that is even comparable with training-based methods. Under the training-free setting, PaintSeg significantly outperforms previous methods by a margin of 4.6 IoU on DUTS-TE and 3.4 IoU on ECSSD. We attribute the performance improvement to the error correction capability of PaintSeg. With alternating between I-step and O-step, the proposed PanintSeg can handle noisy prompts effectively. The robustness of PaintSeg will be discussed in more detail in Section 5.4.
Point prompt. As shown in Table 2, we compare our method with state-of-the-art point prompt segmentation approaches. PaintSeg consistently outperforms the training-free methods. Even compared to training-based methods with ground truth supervision, PaintSeg still achieves the best performance on GrabCut and DAVIS datasets. MIS [33] is an unsupervised approach equipped with second-stage training. We notice that our method can significantly outperform it in terms of IoU, with improvements of 8.2, 6.8, and 16.1 on GrabCut, Berkeley, and DAVIS correspondingly.
Box prompt. Since there is no unsupervised box-prompted segmentation that can be directly compared, we compare the proposed method with several baselines including TokenCut [66], CutLER [65] and MaskRCNN [24]. We first cut off the ground truth box region and then run the baselines. As shown in Table 3, when compared with training-based Mask-RCNN and CutLER, PaintSeg shows suboptimal performance, which can be explained by the lack of training to handle complex
Point Prompt
RITM
PaintSeg
Figure 3: Qualitative results of baselines and our PaintSeg with point and mask prompts. Green point denotes the point prompt. The mask prompt is generated by unsupervised TokenCut [66]. BS represents the bilateral solver [3]. We compare with RITM [58] and TokenCut [66].
Mask Prompt
TokenCut + BS
PaintSeg
I-step	O-step	AC	IoU
✓	78.4
✓	77.9
✓	✓	79.5
✓	✓	✓	80.6
Prompt	0% Noise	15% Noise	30% Noise
Point	60.8	60.4	58.9
Scribble	64.3	63.7	62.7
Box	71.0	70.7	70.1
Coarse Mask	80.6	80.0	79.3
Table 4: Module effectiveness in AMCP. AC: adversarial constraint for mask updating.
Table 5: Prompt robustness. We add random noise to the prompt to evaluate the robustness of AMCP. The noise scale is determined by half the length of the object box diagonal.
scenarios. However, as MaskRCNN is trained on 80 COCO object categories, the "unseen" gap remains substantial. PaintSeg provides an alternative solution that is not reliant on training, thus making it more general and capable of handling new categories of objects. When compared with unsupervised approaches, our method eclipses TokenCut by a large margin on both PASCAL VOC and COCO MVal datasets.
Qualitative results. We visualize the qualitative results with point and coarse mask prompt in Fig. 3. Our visualization depicts comparably reliable results. Comparatively, PaintSeg segments are relatively complete object, while baselines miss some parts of it.
5.4 Analyses
Module effectiveness in AMCP. We step by step add proposed modules in AMCP to validate the effectiveness. As shown in Table 4, we report the results on ECSSD with coarse-mask prompts. We observe that the missing of either step impacts the performance, as evidenced by the significant drop in IoU (compared to alternating I-step and O-step). With the adversarial mask updating constraint, AMCP achieves the best performance of 80.6 IoU.
Robustness of AMCP with different prompts. In Table 5, we add noise to the initial prompt by randomly shifting the position to investigate the robustness of AMCP. The scale of random noise is determined, w.r.t., half the length of the diagonal of the ground-truth bounding box. We observe that AMCP remains robust and only shows a slight performance drop with a noise rate of less than  $30\%$ . The robust capability can be attributed to 1) the alternating I-step and O-step to leverage both background and object shape consistency, and 2) the adversarial mask updating to tackle the background false-positives and foreground false-negatives.
Design choices in AMCP. We conduct experiments to ablate the design choices in AMCP and their impacts on the segmentation performance. We first study the effect of cluster center numbers for quantizing contrastive potential. With a larger cluster center, AMCP will ignore more ambiguous regions. As shown in Table 6a, we notice a cluster center of 2 achieves the best performance for mask prompt. After that, we ablate on the AMCP step number in Table 6b. The segmentation performance keeps increasing until reaching a step number of 5. In this way, we choose 5 as our step number. As we leverage the diffusion-based generative model, we ablate the iterations for the diffusion process as
K	2	3	4
IoU	80.6	72.3	61.5
(a) Cluster center.
T	3	4	5	6
IoU	78.4	79.1	80.6	80.5
(b) Step number.
Iter	10	30	50
IoU	77.3	78.7	80.6
(c) Iter. for painting.
Rate	0.9	1.0	1.1	1.2
IoU	69.2	72.4	80.6	80.0
(d) Box size for contrasting.
Table 6: Design choices for AMCP. We report the performance with the coarse-mask prompt on ECSSD. (a) We ablate the cluster center when contrasting. (b) We ablate the step number for AMCP. (c) We ablate the diffusion iteration for generative painting. (d) We ablate on the cropped box size when contrasting. The rate denotes the proportion of cropped mask and the box of the current stage object mask.
Figure 4: Iterative process of AMCP with box prompt. We inverse the outputted background mask in I-step for better comparison. We only visualize the box prompted area.
it can impact the image quality. As expected, Table 6c demonstrates that a larger iteration number can reach a better performance. To filter out irrelevant background regions, we crop a box region wrapping the given object mask to contrast images. We ablate the box size in Table 6d. We notice that a box slightly larger than the bounding box to the given mask can achieve the best performance. An explanation for this could be that a box tightly enclosing an object will result in a high proportion of object region, which may dominate the features and lead to ambiguity. Properly introducing background can make the extracted features more discriminative and easier for clustering.
Visualization of mask updating. To better illustrate the iterative process of AMCP, as shown in Fig. 4, we visualize the averaged mask output (among  $N$  painted images in each step) for each step with a box prompt. As the given mask only contains background false positives, I-step plays a major role to cut false-positive backgrounds in AMCP. The mask shrink can also be observed after the O-step which is due to the binarization of the averaged mask from the I-step instead of contrastive painting. We observe that the updated masks are gradually closer to the mask of the target object with AMCP.
To conclude, PaintSeg bridges the gap between generative models and segmentation. It is designed to provide a robust and training-free approach to unsupervised image object segmentation. With the proposed adversarial masked contrastive painting (AMCP) process, PaintSeg creates a contrast between the original image and the painted image by alternately applying I-steps ( inpainting) and O-steps (outpainting). The alternating I-step and O-step gradually improve the accuracy of the object mask by leveraging consistency in the background and the shape of the object. The competitiveness of our method on seven different image segmentation datasets suggests that PaintSeg can deal with inaccurate initial masks and adapt to various visual prompts, such as coarse masks, bounding boxes,
scribbles, and points. An extensive ablation analysis indicates a number of key factors and advantages of the proposed model, including its design choices and generalizability.
Limitation. In spite of PaintSeg's high performance for training-free image segmentation with heterogeneous visual prompts, it does not possess object discovery capabilities and therefore cannot automatically recognize instance-level masks in an image. Developing discovery capability can be achieved by conducting second-stage training on the segmentation results generated by PaintSeg, which is our future research focus.
[72] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground-background integration. In European Conference on Computer Vision, pages 332-348. Springer, 2020.
[73] Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[74] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315, 2018.
[75] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[76] Wangbo Zhao, Kai Wang, Xiangxiang Chu, Fuzhao Xue, Xinchao Wang, and Yang You. Modeling motion with multi-modal features for text-based video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11737-11746, 2022.
[77] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In European Conference on Computer Vision, 2022.
[78] Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang, Deyu Meng, and Yefeng Zheng. Interactive segmentation as gaussian process classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5297-5306, 2019.
[79] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiaxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. arXiv preprint arXiv:2111.13792, 2021.
A More Comparison with Mask-RCNN
Mask RCNN
Mask RCNN
Figure A: Comparison with Mask RCNN with objects beyond 80 COCO categories.
We present more results compared with supervised Mask RCNN [24]. As shown in Fig. A, we compare box-prompted segmentation with Mask RCNN on objects beyond 80 COCO categories. In the shown examples, we observe that Mask RCNN has difficulty segmenting the correct shape of the object. Instead, PaintSeg provides more accurate object segmentation. As Mask RCNN is only trained on 80 COCO object categories, there is still a substantial gap between the seen and the unseen. In contrast, PaintSeg is a solution that does not require training, which makes it more general and capable of handling new object categories.
B More Ablation Experiments
In this section, we provide additional ablation studies to illustrate the design choices of PaintSeg.
N	1	2	3	4	5	6
IoU	78.8	79.2	79.6	80.1	80.6	80.8
B.1 Sampling Number for Each Step
We average  $N$  painted images in each step to obtain the final mask prediction due to the randomness of the generative painting model. We present an ablation study to illustrate the impact of the number of painted images in each step. As shown in Table A, we report the performance on the ECSSD [54] dataset with coarse mask prompt from TokenCut [66]. We notice that the performance gradually improved with more painted images averaged in each step. As there is no significant difference in performance between five or six painted images used, we set the number of painted images to five in the PaintSeg process.
Table A: Ablation study on the painted image number  $N$  for each step.
DINO [9] VIT-S/8	DINO-V2 [46] VIS-S/14
80.6	80.0
Table B: Ablation study on image projector  $\mathcal{E}$  used in AMCP.
B.2 Image Projector
We conduct an ablation study on image projector  $\mathcal{E}$  as illustrated in Table B. We compare the widely used DINO [9] VIT-S/8 and the latest DINO [46] VIS-S/14. The results demonstrate that DINO with a small patch size achieves better performance. It follows that we consider a smaller patch size since PaintSeg requires fine-grained visual information. A larger patch size will blur the object boundary, resulting in a performance drop.
C More Potential Application
In this section, we discuss more potential applications of PaintSeg beyond prompt-guided object segmentation.
Reassembled Image
Figure B: Potential application in image edition and amodal segmentation. PaintSeg can step-by-step remove objects in the image by using the painted image in I-step. With the segmented object and painted image without objects, we can freely assemble them into a new image. Further, PaintSeg supports amodal segmentation, with the painting capability enabling segmentation of the occluded areas.
C.1 Image Edition
In the I-step of AMCP, the painted image will remove the target object while keeping all other contents in the image. In this way, with the segmented objects and an image without target objects, we can reassemble them into a new image as shown in Fig. B.
C.2 Amodal Segmentation
As shown in Fig. B, PaintSeg can layer-by-layer segment objects. By using the painted image in I-step as the input to the next iteration, PaintSeg can attach the amodal capability. We notice that the bench is occluded by the men in Fig. B. With the PaintSeg, the full shape of the bench can be segmented.
D More Discussion about PaintSeg
D.1 Point Prompt
As depicted in Fig 2, the object mask can be derived by contrasting in/outpainted images with the original ones. In the I-step, there is a substantial difference (Fig 2 (a)) in the object area. In contrast, in the O-step, the object area has a small difference (Fig 2 (b)).
To incorporate the point information, we introduce a prompt term  $\Phi_{prompt}$  in the contrastive potential to enhance the difference based on the point location. Specifically,  $\Phi_{prompt}[i,j] = \mathcal{G}(x,y)[i,j]$  (Equ 2) represents a gaussian offset. Here,  $(x,y)$  refers to the point prompt, serving as the mean, while  $\frac{1}{10}$  of the bounding box's height and width determine the variance (Line 159). During the I-step, we add  $\Phi_{prompt}$  to the contrastive potential to amplify the difference in the prompted area. Conversely, during the O-step, we subtract  $\Phi_{prompt}$  to reduce the difference in the prompted area. This way, when we binary classify the contrastive potential (Equ 3), the point-prompted area will be classified as part of the object area.
D.2 Adversarial Mask Updating
As discussed in Sec 4.1 and Fig 2, the I-step is capable of handling false positives (FP) and the O-step for false negatives (FN). If no mask updating constraints are applied, as the mask updating may not be accurate in the first several steps, new FP can be introduced in I-step. However, the new FP cannot
be addressed in the following O-step and has to wait for the next I-step to be handled. In contrast, if we constrain the I-step only shrinks the object mask, no FP will be introduced in the I-step and the remaining FN can be addressed in the following O-step. Similarly, we constrain O-step only to expand the object mask thus no FN will be introduced. We refer to the "shrink" in I-step and "expand" in O-step as adversarial mask updating. By employing the adversarial mask updating, we can address errors (FP/FN) that occurred during each step immediately without degradation.
D.3 Expectation-Maximization
In PaintSeg, we introduce a latent variable  $I_{paint}$  which is characterized by an off-the-shelf generative model  $p(I_{paint}|I \circ M)$  conditioned on an image  $I$  and a mask  $M$ . represents Hadamard product. In our method, we leverage the AMCP process to estimate and convert the latent variable  $I_{paint}$  into mask prediction  $M$  with alternating I-step and O-step. Mathematically, both I-step and O-step can be formulated as an expectation-maximization-like process.
- Expectation: We introduce a latent variable  $I_{paint}$  in the proposed PaintSeg which is modeled by an off-the-shelf generative painting model  $p(I_{paint}|I \circ M)$ . We assume the generative model will pick the most likely outcome  $I_{paint}$  given  $I$  and  $M$  for every step.
- Maximization: After obtaining the latent variable  $I_{paint}$ , we define a contrastive potential  $\Phi$  and utilize clustering to binarize the mask. Mathematically, the contrasting and clustering processes maximize a posteriori probability  $p(M|I_{paint},I) = e^{-\frac{1}{||M||_0}\Phi (I_{paint},I,M)}$ .
Although we term I-step and O-step separately, they can be formulated as the same EM process. PaintSeg advances the predicted mask to the ground truth by iteratively conducting the EM process in each step.
D.4 Contrastive Potential
There are three terms in the contrastive potential  $\Phi_{paint}$ ,  $\Phi_{color}$  and  $\Phi_{prompt}$ :
-  $\Phi_{paint}$ : As analyzed in Sec 4.1, we can segment the object by contrasting the original and in/outpainted images. In this way, we formulate  $\Phi_{paint}$  to describe the difference between the original and in/outpainted images and maximize it to locate the object.
-  $\Phi_{color}$ : As an image projector, i.e., DINO, is leveraged, the object segmentation from solely  $\Phi_{paint}$  is achieved at a patch level. Therefore, we further introduce a color potential  $\Phi_{color}$  to incorporate pixel-level color information to better segment the object boundary.  $\Phi_{color}$  describes the probability of a pixel belonging to the foreground or background region based on the color similarity. By combining  $\Phi_{paint}$  and  $\Phi_{color}$ , we can achieve high-quality object segmentation with AMCP.
-  $\Phi_{prompt}$ : In addition, the  $\Phi_{prompt}$  is introduced to provide additional location information to guide the PaintSeg to locate the target object. We will add the discussion in the revision
E Difference with Previous Segmentation Approaches
In this section, we discuss the major differences between the proposed PaintSeg and previous object segmentation methods as follows.
Discriminative v.s. Generative + Discriminative. Conventional object segmentation is a discriminative task that leverages a neural network  $\theta$  to model the conditional probability of the object mask  $M$  given the image  $I$  as condition  $p_{\theta}(M|I)$ . In PaintSeg, we have mask, paint, and contrast operations in each step. Specifically, in paint operation, we enroll a generative model to estimate painted image  $I_{paint}$  with mask  $M$  and image  $I$  as conditions. After that, the mask can be obtained by comparing the generated image with the original one with a contrastive potential  $\Phi$ . As discussed in Section D.3, the paint operation is a generative process to estimate latent variable  $p(I_{paint}|I \circ M)$  and the contrast operation is a discriminative process to obtain a mask prediction based on  $p(M|I_{paint}, I)$ . PaintSeg achieves training-free by constructing a bridge to generative painting models which permits object shape consistency and background content consistency.
Pixel v.s. Pixel difference. Conventional object segmentation leverages a network to project an image to the feature space and then binarize (cluster) each pixel into foreground or background classes. Differently, instead of directly clustering over the input image, PaintSeg utilizes the difference between the painted and original image, as a proxy, to leverage the object shape prior and background consistency. The contrastive scheme is rooted in the decomposable nature of images and paves a way to incorporate generated images to segment objects.
Training v.s. Training-free. Conventional object segmentation approaches train the neural network to segment objects requiring time-consuming and expensive data labeling. Some unsupervised segmentation methods [4, 10, 5, 61] find a segment from a generative model while they typically require training a network on top of the generative model. Instead, our method is a training-free unsupervised method that learns to segment objects from a generative painting model. We consider the PaintSeg provides a way to bridge the generative model and segmentation which may inspire future research.
F Failure Case Analysis
We analyze the failure case here. As shown in Fig. C, we visualize a failure case when using a point as the prompt. We notice the adjacent car is segmented as a false positive, which is due to the semantic and visual similarity between the target and false positive cars. Despite our method is capa
With Point Prompt
Figure C: Illustration of failure case.
With Box Prompt
With Point Prompt
ble of handling multiple objects with point prompt (right of Fig. C), crowded scenarios can make it difficult to segment the accurate object boundary. However, the issue can be overcome through box prompt.
G More Visualization
In this section, we demonstrate more visualization of PaintSeg. We show more qualitative results with box prompt in Fig. D, with point prompt in Fig. E and with coarse mask prompt in Figs. F and G.
Figure D: More visualization of PaintSeg with box prompt on COCO MVal.
Figure E: More visualization of PaintSeg with point prompt. The point prompt is illustrated by the red point on the image on DAVIS and Berkeley and GrabCut.
Figure F: More visualization of PaintSeg with coarse mask prompt on ECSSD.
Figure G: More visualization of PaintSeg with coarse mask prompt on ECSSD.