SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiang Li $^{1}$ , Chung-Ching Lin $^{2}$ , Yinpeng Chen $^{2}$ , Zicheng Liu $^{2}$ , Jinglu Wang $^{2}$ , Rita Singh $^{1}$ , Bhiksha Raj $^{1,3}$
$^{1}$ CMU,  $^{2}$ Microsoft,  $^{3}$ MBZUAI
xl6@andrew.cmu.edu
The paper introduces PaintSeg, a new unsupervised method for segmenting objects without any training. We propose an adversarial masked contrastive painting (AMCP) process, which creates a contrast between the original image and a painted image in which a masked area is painted using off-the-shelf generative models. During the painting process, inpainting and outpainting are alternated, with the former masking the foreground and filling in the background, and the latter masking the background while recovering the missing part of the foreground object. Inpainting and outpainting, also referred to as I-step and O-step, allow our method to gradually advance the target segmentation mask toward the ground truth without supervision or training. PaintSeg can be configured to work with a variety of prompts, e.g. coarse masks, boxes, scribbles, and points. Our experimental results demonstrate that PaintSeg outperforms existing approaches in coarse mask-prompt, box-prompt, and point-prompt segmentation tasks, providing a training-free solution suitable for unsupervised segmentation. Code: https://github.com/1xa9867/PaintSeg.
With deep learning advancements, significant progress has been made in the field of image generation and segmentation in recent years. A particular generative model, the denoising diffusion probabilistic model (DDPM), has demonstrated outstanding performance in a variety of generative tasks, such as image inpainting [49, 16, 34] and text-to-image synthesis [21, 20, 79]. Similar developments have occurred in the field of object segmentation, such as the strong zero-shot capability and excellent segmentation quality demonstrated by SAM [30].
Image generation and segmentation can be mutually beneficial. Segmentation has been shown to be a critical technique in improving the realism and stability of generative models by providing pixel-level guidance during the synthesis process [75, 29]. Interesting to note is the fact that the relationship between segmentation and generative models does not appear to be solely one-sided. Generative models learning to "paint" objects actually know where the painted object is. The emergence of unsupervised image segmentation methods utilizing generative adversarial networks (GANs) has produced a line of methods that can segment objects in images [4, 10, 5] using generative models. These methods work on the assumption that object appearance and location can be perturbed without compromising scene realism. By using the GAN architecture to discriminate between perturbed and real images, these methods can achieve effective object segmentation. Moreover, a follow-up work [61] develops an approach to leverage pre-trained GAN by identifying "segmenting" direction in the latent space to discriminate object shapes.
In this paper, we present PaintSeg, an approach for unsupervised image segmentation that leverages off-the-shelf generative models. Unlike previous methods [61, 4] that require training on top of these models, PaintSeg introduces a novel, training-free segmentation approach that relies on an adversarial masked contrastive painting (AMCP) process. The AMCP process creates a contrast
Figure 1: Illustration of adversarial masked contrastive painting (AMCP). Given an input image and an initial mask, AMCP leverages alternating I-step and O-step to gradually refine the segmentation mask until it converges to the ground truth. Both steps share the same mask, paint, and contrast operations. The updated mask in each step is achieved by binarizing the contrastive difference between the original and painted images.
between the original image and a painted image by alternating between inpainting and outpainting, with the former filling in the background and masking the foreground, and the latter retrieving the missing part of the object while masking the background and a portion of the foreground.
Both steps, as shown in Fig. 1, share the same operations while taking input from background and foreground masks, correspondingly. In the I-step, the object region is removed from the painted image, creating a significant contrast with the original image. Conversely, in the O-step, the background region exhibits a remarkable difference between the original and painted image. The foreground or background mask can be obtained by binarizing the contrastive difference in each step.
Although either I-step or O-step is capable of discriminating objects, the single-step method is less robust. The I-step involves segmenting objects based on background consistency without taking into account object information. As a result, the segmentation may be imperfect if the object part resembles the background. Similarly, in the O-step, only the object shape prior is utilized, resulting in a lack of background knowledge. This problem is addressed by introducing adversarial mask updating, in which I-steps and O-steps are alternated. During I-step, we only shrink the object mask to cut off background false positives, while during O-step, we expand it to link up foreground false negatives. Thereby, even if errors occur during the iteration of AMCP, they will be corrected in the next step without degradation. With the adversarial mask updating, the target mask can be gradually advanced to the ground truth.
With the robustness of AMCP, PaintSeg can deal with inaccurate initial masks and adapt to various visual prompts, such as coarse masks, bounding boxes, scribbles, and points. Compared to the recently published successes in image object segmentation study, our main contributions are as follows:
- We propose PaintSeg, a training-free approach to segmenting image objects based on heterogeneous visual cues. The method provides a direct bridge between generative models and segmentation.
- We introduce adversarial masked contrastive painting (AMCP), consisting of alternating I-step and O-step, to robustly segment objects.
- We conduct extensive experiments for analysis and comparisons on seven different image segmentation datasets, the results of which show the superiority and generalization ability of our methods.
2.1 Unsupervised Image Segmentation
Unsupervised methods for image segmentation are extensively investigated with the advancements in self-supervised. DINO [8] provides a self-supervised approach to explicitly bring out underlying semantic segmentation of images using a Vision Transformer (ViT) [17]. Based on DINO, LOST [56], Deep Spectral Methods [44] and TokenCut [66] leverage self-supervised ViT features and propose to
segment objects using NCut [52]. Subsequently, [57, 55] introduce a second-stage training approach to further improve the segmentation quality. Found [57] incorporates background similarity as an additional refinement factor, while SelfMask [55] utilizes an ensemble of features [7, 8, 12] to enhance image representation. CutLER [65] enables multiple objects discovery capability by iteratively cutting objects with NCut and introduces a more powerful second-stage training. FreeSOLO [64] generates coarse masks with correlation maps that are then ranked and filtered by a "maskness" score. Another line of unsupervised methods learns to generate a realistic image by combining a foreground, a background and a mask [5, 71, 70, 59, 32, 18, 25] and then the object segmentor can be obtained as a byproduct.
2.2 Prompt-guided Segmentation
Prompt-guided segmentation aims to segment objects assigned by prompts, e.g., mask, box, scribble and point. Semi-supervised video object segmentation (VOS) [1, 68, 38], aiming at segmenting object masks across frames given the first frame mask, is a typical mask-prompt task. The mainstream of VOS methods [72, 73] constructs pixel-level correspondence and propagates masks by exploring matches among adjacent frames. Interactive segmentation (IS) [78, 40, 58, 23] is another line of prompt-guided segmentation. IS permits users to leverage scribbles and points to assign target objects and segment them. In addition, an interactive correction is also featured by IS which introduces additional prompts to correct misclassified regions. MIS [33] is a recent work tackling unsupervised IS and proposes a multi-granularity region proposal generation to refine the mask. SAM [30] is a recently introduced zero-shot method for prompt-based segmentation which introduces a large-scale dataset and a strategy to mitigate the ambiguity of prompt. Beyond visual prompts, objects can also be referred by natural language or acoustic prompts. Referring image segmentation (RIS) [28, 74] and referring video object segmentation (R-VOS) [11, 76, 15, 35, 37] aims to segment objects in image/video referred by linguistic expressions. Audiovisual segmentation [77, 36] aims to segment sound sources in the given audiovisual clip.
2.3 Conditional Image Generation
Conditional image generation refers to the process of generating images based on specific conditions or constraints. In most instances, the condition can be based on class labels, partial images, semantic masks, etc. Cascaded Diffusion Models [26] uses ImageNet class labels as a condition to generate high-resolution images with a two-stage pipeline of multiple diffusion models. [51] guides diffusion models to produce novel images from low-density regions of the data manifold. Apart from these, CLIP [48] has been widely used in guiding image generation in GANs with text prompts [21, 20, 79]. For diffusion models, Semantic Diffusion Guidance [41] investigates a unified framework for diffusion-based image generation with language, image, or multi-modal conditions. Dhariwal et al. [14] apply an ablated diffusion model to use the gradients of a classifier to guide the diffusion with a trade-off between diversity and fidelity. Additionally, Ho et al.[27] introduce classifier-free guidance in conditional diffusion models by mixing the score estimates of a conditional diffusion model and a jointly trained unconditional diffusion model.
We tackle the unsupervised prompt-guided image object segmentation task, which aims to predict the object mask  $M \in \{0,1\}^{1 \times H \times W}$  in an image  $I \in \mathbb{R}^{3 \times H \times W}$  given a visual prompt  $P \in \{0,1\}^{1 \times H \times W}$ . The visual prompt can have a format of a point, a scribble, a bounding box or a coarse mask of the target object  $P \in \{P_{point}, P_{scrib}, P_{box}, P_{mask}\}$ . Following the convention, we assume the ground-truth object mask  $M$  must have an overlap with the visual prompt  $P \cap M \neq \emptyset$ .
4 Adversarial Masked Contrastive Painting
PaintSeg leverages adversarial masked contrastive painting (AMCP) to gradually refine the initial prompt  $P$  to the object mask  $M$ . The AMCP approach is composed of alternating I-steps and O-steps, as illustrated in Figure 1. During each step, a region of the image is masked out based on the previous iteration's mask, and the masked region is then repainted and compared to the original image to refine the mask prediction. To improve the segmentation's robustness, PaintSeg introduces adversarial
(a) I-step given mask with FP
(b) O-step given mask with FN
Figure 2: Illustration of I-step and O-step with initial prompts. (a) We show an I-step with a box prompt as the initial mask, where the object region has a significant difference between the original and painted images. (b) We show an O-step with a scribble prompt as the initial mask, where the object region has a small difference between the original and painted images.
mask updating, which helps to ensure that the mask accurately reflects the object's boundaries. The I-step is used to shrink the object mask by leveraging background consistency, thereby eliminating false-positive regions. On the other hand, the O-step expands the object mask by utilizing object shape consistency to link up false-negative foreground regions.
4.1 Contrastive Painting
We first discuss the rationality of segmenting objects by contrasting painted and original images. Given a visual prompt  $P$ , the relation between the prompted area and object mask can be categorized into three types: background false positive (Fig. 2 (a)), foreground false negative (Fig. 2 (b)) and a hybrid of both. We tackle the prompt-guided segmentation by separately addressing the background false positive and foreground false negative with I-step and O-step respectively.
We discuss the painted content with different mask situations. To avoid ambiguity, we first denote the generative model taking background and foreground as conditions as inpainting model  $\phi(\cdot)$  outpainting model  $\psi(\cdot)$  respectively. We consider the prompted area as the initial mask  $M_0 = P$ . When the initial mask has false positives, i.e.,  $M \subset M_0$ , as shown in Fig. 2 (a), the inpainted content tends to complete the background based on the background consistency. In this way, the inpainted pixels inside the object will have a significant difference compared to the original image. In contrast, when the initial mask has false negatives, i.e.,  $M_0 \subset M$ , as shown in Fig. 2 (b), the outpainted content tends to complete the partial object leading to a low difference with the original image inside object region. We notice that I-step can address background false-positive and O-step can address foreground false-negative. By alternating conducting I-step and O-step, we can leverage both foreground and background consistency and address more complicated cases.
4.2 Contrastive Potential
Given the a image  $I$  and a mask  $M_{t}$ , we define a contrastive potential  $\Phi$  to measure the region relations, which contains three terms
$$
\Phi = \lambda_ {p a i n t} \Phi_ {p a i n t} + \lambda_ {c o l o r} \Phi_ {c o l o r} + \lambda_ {p r o m p t} \Phi_ {p r o m p t}. \tag {1}
$$
We introduce a box region  $B$  that encloses the foreground and define the  $\Phi_{paint}$  term as the distance between the painted and the original image. Specifically,  $\Phi_{paint} = B \circ |\mathcal{E}(I) - \mathcal{E}(I_{paint})|_2$ , where  $\mathcal{E}: \mathbb{R}^{3 \times H \times W} \to \mathbb{R}^{C \times H \times W}$  is a function that projects the image to a high-dimensional space.  $\circ$  denotes the Hadamard product.
The  $\Phi_{color}$  term measures the pixel-level color similarity inside and outside the mask  $M_t$ . To compute it, we use the output of the conditional random field algorithm [31] and define  $\Phi_{color} = \mathcal{C}(M_t \circ B, I \circ B)$ , where  $\mathcal{C}$  is a function that takes a mask and an image as inputs and outputs the probability of whether a pixel should belong to the masked region.
To further incorporate prompt information, we introduce prompt priors  $\Phi_{prompt}$  to the contrastive potential for box, scribble, and point prompts. Let us denote  $[x_l,y_l]$  as the coordinates of the  $l$ -th
point in the prompt area. The prompt prior is defined as follows:
$$
\Phi_ {p r o m p t} [ i, j ] = \max  _ {l} \mathcal {G} \left(x _ {l}, y _ {l}\right) [ i, j ], \tag {2}
$$
where  $\mathcal{G}$  is a two-dimensional Gaussian function, and  $\mathcal{G}[i,j] = \exp \left(\frac{(i - x_l)^2}{\sigma_x^2} +\frac{(j - y_l)^2}{\sigma_y^2}\right)$ . Specifically, for the box prompt, we only take the center point of the box into account. The prompt priors are designed to leverage the positional information of the prompts to better locate the target object. By taking the maximum value of the Gaussian function over all points in the prompt area,  $\Phi_{prompt}$  captures the overall strength of the prompt signal.
4.3 Adversarial I/O-Step
As shown in Fig. 1, I-step and O-step share the same mask, paint, and contrast processes while the input mask in I-step is the background mask and, in O-step, the foreground mask. Given the original image  $I$  and an input mask  $M_{t}$  from the  $t$ -th step of AMCP (assuming  $M_{t}$  is a background mask thus  $t + 1$ -th step is an I-step), we first filter out the masked region by  $I \circ M_{t}$  and then paint the image  $I_{paint} = \phi(I \circ M_{t})$ . As discussed in Section 4.1, the foreground region will have a significant difference between painted and original images. We obtain the updated mask  $M_{t + 1}$  by k-means clustering  $\mathcal{K}(\cdot)$  over the contrastive potential  $\Phi$ . Let us denote  $\mu_{k}$  and  $S_{k} \in \{0,1\}^{H \times W}$  as the average value of all samples in the  $k$ -th cluster and its corresponding identity map ( $S_{k}[i,j] = 1$  if pixel  $[i,j]$  belongs to center  $k$  else 0). The updated mask can be found by
$$
M _ {t + 1} = S _ {k ^ {*}}, k ^ {*} = \arg \max  _ {k} \mu_ {k}. \tag {3}
$$
The updated mask  $M_{t + 1}$  is a foreground mask thus the next step will be an O-step. Similarly, we paint the image by  $I_{paint} = \psi (I\circ M_{t + 1})$ . Here, the difference in background area will have a significant difference between painted and original images. Thereby, the updated mask  $M_{t + 2}$  from O-step can be computed using the same rule as Eq. (3) which leads to a background mask. By updating the mask by Eq. (3), we notice that when the input mask  $M_t$  is a background mask, then the output mask will be a foreground mask and vice versa. Thereby, the alternating I-step and O-step can be automatically achieved.
As discussed in Section 4.1, I-step is advantageous for reducing false positives in the background, whereas O-step is beneficial for reducing false negatives in the foreground. Specifically, the updated mask is configured to only cut off pixels in the I-step, and to only link up pixels in the O-step. Let  $M_t^+$  and  $M_t^-$  as the dilated and eroded masks of  $M_t$ . We constrain to only update the regions near the foreground-background boundary. In this way, the updating rule for AMCP can be rewritten as
$$
M _ {t + 1} = \left\{ \begin{array}{l l} S _ {k ^ {*}} \circ \Delta^ {-} + \bar {M} _ {t} \circ (1 - \Delta^ {-}), & \text {I - s t e p} \\ S _ {k ^ {*}} \circ \Delta^ {+} + \bar {M} _ {t} \circ (1 - \Delta^ {+}), & \text {O - s t e p} \end{array} , k ^ {*} = \arg \max  _ {k} \mu_ {k} \right. \tag {4}
$$
where  $\Delta^{-} = \bar{M}_{t} - \bar{M}_{t}^{-}$  and  $\Delta^{+} = M_{t}^{+} - M_{t}$  are the inner and outer neighbors of  $M_{t}$ . Through the adversarial alternation of I-steps and O-steps, AMCP can handle more complex cases involving both false positives and false negatives. Due to the randomness inherent in generative painting, we paint the image  $N$  times in each step, and use the averaged mask as an output.
In this section, we introduce the mathematical formulation of AMCP. Mathematically, an image can be represented as a masked combination of a foreground image  $I_{F}$  and a background image  $I_{B}$
$$
I = I _ {F} \circ M + I _ {B} \circ \bar {M}, M \in \{0, 1 \} ^ {H \times W \times 1}. \tag {5}
$$
$M$  is a foreground mask.  $\bar{M} = 1 - M$ . An inpainting model  $\phi[\cdot]$  is defined to generate pixels inside the mask given the pixels outside the mask as a condition. Similarly, an outpainting model  $\psi[\cdot]$  predicts pixels outside the mask given the pixels inside the mask as a condition. In our method, we aim to find a  $M$  that maximizes
$$
\underset {M} {\arg \max } \underbrace {\left\| I \circ \Delta^ {-} - \phi (I \circ \bar {M}) \circ \Delta^ {-} \right\| _ {d}} _ {\text {I - S t e p}} + \underbrace {\left\| I \circ \Delta^ {+} - \psi [ I \circ M ] \circ \Delta^ {+} \right\| _ {d}} _ {\text {O - S t e p}} \tag {6}
$$
Method	Training	DUTS-TE [62]	ECSSD[54]
— Compared to methods with training —
SelfMask [55] CVPRW22	✓	62.6	78.1
SelfMask [55] CVPRW22 + BS [2]	✓	66.0	81.8
FOUND [57] CVPR23	✓	63.7	79.3
FOUND [57] CVPR23 + BS [2]	✓	66.3	80.5
PaintSeg	67.0	80.6
— Compared to methods without training —
Melas-Kyriazi et al. [43] ICLR22	52.8	71.3
LOST [56] BMVC21	51.8	65.4
LOST [56] BMVC21 + BS [2]	57.2	72.3
DSS [45] CVPR22	51.4	73.3
TokenCut [66] CVPR22	57.6	71.2
TokenCut [66] CVPR22 + BS [2]	62.4	77.2
SelfMask† [55] CVPRW22	46.6	64.6
FOUND† [57] CVPR23	-	71.7
PaintSeg	67.0	80.6
Table 1: Quantitative results of coarse mask-prompted segmentation on DUTS-TE and ECSSD. PaintSeg utilizes the coarse mask generated by unsupervised TokenCut [66] as prompt. BS denotes the application of the post-processing bilateral solver on the generated masks and the column 'Learning' specifies which methods have a training step. The best result per section is highlighted in bold. The second best result for each section is underlined.  $\dagger$  indicates the first-stage pseudo mask obtained without training.
The first term aims to maximize the difference between the original image  $I$  and the inpainted image  $\phi(I \circ \bar{M})$  in the inner neighbor  $\Delta^{-}$  which corresponds to the I-step in AMCP. The second term aims to maximize the difference between the original image  $I$  and the outpainted image  $\psi[I \circ M]$  in the outer neighbor  $\Delta^{+}$  corresponding to the O-step.
In each step, our mask, paint, and contrast operations can be considered as an expectation-maximization-like (EM-like) process with the latent variable of  $I_{paint}$  to maximize Eq. (6). On one hand, the  $I_{paint}$  is estimated by the mask and paint operations where the conditional probability  $p(I_{paint}|I,M)$  is characterized by the generative painting models (expectation step). On the other hand, the predicted mask  $M$  can be updated by maximizing the contrastive potential  $\Phi$  (maximization step). Since the EM algorithm is sensitive to the initial value, solely updating with I-step or O-step cannot achieve robust performance. With the alternating I-step and O-step, we introduce an adversarial updating process which leads to a more robust mask estimation.
For mask-prompt segmentation, we evaluate on DUTS-TE [63] and ECSSD [53]. DUTS-TE contains 5,019 images selected from the SUN dataset [67] and ImageNet test set [13]. ECSSD [53] contains 1,000 images that were selected to represent complex scenes. For box-prompt segmentation, we evaluate on PASCAL VOC [19] val set and COCO [39] MVAL datasets. COCO MVal contains 800 object instances from the validation set with 10 images from each of the 80 categories. For point-prompt segmentation, we use three datasets including GrabCut [50] which contains 50 images and corresponding segmentation masks that delineate a foreground object; Berkeley [42] which contains 96 images with 100 instances with more difficulty than GrabCut and DAVIS [47] which is a video dataset and  $10\%$  of the annotated frames are randomly selected, yielding 345 images that are used in the evaluation
Evaluation metrics. In accordance with previous methods [30, 66], we evaluate segmentation quality using intersection over union (IoU).
Implementation details. We leverage the inpainting models trained with latent-diffusion pipeline [49] as our  $\phi$  and  $\psi$ . We set the diffusion iterations to 50. We leverage DINO [8] pretrained VIT-S/8
Method	Training	Supervision	GrabCut	Berkeley	DAVIS
— Compared to methods with training —
DIOS[69]CVPR16	✓	✓	64.0	66.0	57.8
RITM [58]ICIP22	✓	✓	81.0	77.7	66.0
MIS [33]arXiv23	✓	76.2	63.2	53.3
PaintSeg	84.4	70.0	69.4
— Compared to methods without training —
Random Walk [22]TPAMI06	25.7	26.2	<20
GrowCut [60]GraphiCon05	26.7	26.2	-
GraphCut [6]ICCV01	41.8	33.9	<20
PaintSeg	84.4	70.0	69.4
Table 2: Quantitative comparison of point-prompted segmentation on GrabCut, Berkeley, and DAVIS. The point prompt is given as the centroid of each object.
Method	Training	Supervision	PASCAL VOC	MVal
- Compared to methods with training —
Mask-RCNN [24]ICCV17	✓	✓	73.2	79.4
CutLER [65]CVPR23	✓	63.5	74.8
PaintSeg	59.7	69.6
- Compared to methods without training —
TokenCut [66]CVPR22	30.2	34.7
PaintSeg	59.7	69.6
Table 3: Quantitative comparison of box-prompted segmentation on PASCAL VOC and COCO MVal.