Title: Label-efficient Segmentation via Affinity Propagation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computer Vision

Segmentation

Weakly-supervised Learning
Abstract: Weakly-supervised segmentation with label-efficient sparse annotations has attracted increasing research attention to reduce the cost of laborious pixel-wise labeling process, while the pairwise affinity modeling techniques play an essential role in this task. Most of the existing approaches focus on using the local appearance kernel to model the neighboring pairwise potentials. However, such a local operation fails to capture the long-range dependencies and ignores the  topology of objects. In this work, we formulate the affinity modeling as an affinity propagation process, and propose a local and a global pairwise affinity terms to generate accurate soft pseudo labels. An efficient algorithm is also developed to reduce significantly the computational cost. The proposed approach can be conveniently plugged into existing segmentation networks. Experiments on three typical label-efficient segmentation tasks, i.e. box-supervised instance segmentation, point/scribble-supervised semantic segmentation and CLIP-guided semantic segmentation, demonstrate the superior performance of the proposed approach.

Label-efficient Segmentation via Affinity Propagation

Abstract

Weakly-supervised segmentation with label-efficient sparse annotations has attracted increasing research attention to reduce the cost of laborious pixel-wise labeling process, while the pairwise affinity modeling techniques play an essential role in this task. Most of the existing approaches focus on using the local appearance kernel to model the neighboring pairwise potentials. However, such a local operation fails to capture the long-range dependencies and ignores the topology of objects. In this work, we formulate the affinity modeling as an affinity propagation process, and propose a local and a global pairwise affinity terms to generate accurate soft pseudo labels. An efficient algorithm is also developed to reduce significantly the computational cost. The proposed approach can be conveniently plugged into existing segmentation networks. Experiments on three typical label-efficient segmentation tasks, i.e. box-supervised instance segmentation, point/scribble-supervised semantic segmentation and CLIP-guided semantic segmentation, demonstrate the superior performance of the proposed approach.

1 Introduction

Segmentation is a widely studied problem in computer vision, aiming at generating a mask prediction for a given image, e.g., grouping each pixel to an object instance ( instance segmentation ) or assigning each pixel a category label ( semantic segmentation ). While having achieved promising performance, most of the existing approaches are trained in a fully supervised manner, which heavily depend on the pixel-wise mask annotations, incurring tedious labeling costs [1]. Weakly-supervised methods have been proposed to reduce the dependency on dense pixel-wise labels with label-efficient sparse annotations, such as points [2-4], scribbles [5-7], bounding boxes [8-11] and image-level labels [1215]. Such methods make dense segmentation more accessible with less annotation costs for new categories or scene types.

Most of the existing weakly-supervised segmentation methods [16, 13, 8, 17, 3, 10] adopt the local appearance kernel to model the neighboring pairwise affinities, where spatially nearby pixels with similar color ( i.g. , LAB color space [8, 3] or RGB color space [13, 16, 17, 10]) are likely to be in the same class. Though having proved to be effective, these methods suffer from two main limitations. First, the local operation cannot capture global context cues and capture long-range affinity dependencies. Second, the appearance kernel fails to take the intrinsic topology of objects into account, and lacks capability of detail preservation.

To address the first issue, one can directly enlarge the kernel size to obtain a large receptive filed. However, this will make the segmentation model insensitive to local details and increase the computational cost greatly. Some methods [14, 12] model the long-range affinity via random walk [18], but they cannot model the fine-grained semantic affinities. As for the second issue, the tree-based

box-supervised instance segmentation

point/scribble-supervised semantic segmentation

annotation-free semantic segmentation

input image

supervise

pseudo label GT

GT

supervise

(a)

(b)

or

CLIP

supervise

(c)

or

input image

input image

pseudo label

refined

prediction

prediction

prediction

pseudo label

APro

APro

APro

approaches [7, 19] are able to preserve the geometric structures of objects, and employ the minimum spanning tree [20] to capture the pairwise relationship. However, the affinity interactions with distant nodes will decay rapidly as the distance increases along the spanning tree, which still focuses on the local nearby regions. LTF-V2 [21] enables the long-range tree-based interactions but it fails to model the valid pairwise affinities for label-efficient segmentation task.

With the above considerations, we propose a novel component, named Affinity Propagation ( APro ), which can be easily embedded in existing methods for label-efficient segmentation. Firstly, we define the weakly-supervised segmentation from a new perspective, and formulate it as a uniform affinity propagation process. The modelled pairwise term propagates the unary term to other nearby and distant pixels and updates the soft pseudo labels progressively. Then, we introduce the global affinity propagation, which leverages the topology-aware tree-based graph and relaxes the geometric constraints of spanning tree to capture the long-range pairwise affinity. With the efficient design, the O ( N 2 ) complexity of brute force implementation is reduced to O ( N log N ) , and the global propagation approach can be performed with much less resource consumption for practical applications. Although the long-range pairwise affinity is captured, it inevitably brings in noises based on numerous pixels in a global view. To this end, we introduce a local affinity propagation to encourage the piece-wise smoothness with spatial consistency. The formulated APro can be embedded into the existing segmentation networks to generate accurate soft pseudo labels online for unlabeled regions. As shown in Fig. 1, it can be seamlessly plugged into the existing segmentation networks for various tasks to achieve weakly-supervised segmentation with label-efficient sparse annotations.

We perform experiments on three typical label-efficient segmentation tasks, i.e. box-supervised instance segmentation, point/scribble-supervised semantic segmentation and annotation-free semantic segmentation with pretrained CLIP model, and the results demonstrated the superior performance of our proposed universal label-efficient approach.

2 Related Work

Label-efficient Segmentation. Label-efficient segmentation, which is based on the weak supervision from partial or sparse labels, has been widely explored [1]. Different from semi-supervised settings [22, 23], this paper mainly focuses on the segmentation with sparse labels. In earlier literature [24-28], it primarily pertained to image-level labels. Recently, diverse sparse annotations have been employed, including point , scribble , bounding box , image-level label and the combinations of them. We briefly review the weakly-supervised instance segmentation, semantic segmentation and panoptic segmentation tasks in the following.

For weakly-supervised instance segmentation, box-supervised methods [16, 8, 17, 9, 29, 30, 11, 10, 31] are dominant and perform on par with fully-supervised segmentation approaches. Besides, the 'points + bounding box' annotation can also achieve competitive performance [32, 33]. As for weakly-supervised semantic segmentation, previous works mainly focus on the point-level supervision [2, 34, 6] and scribble-level supervision [5, 35, 36], which utilize the spatial and color information of the input image and are trained with two stages. Liang et al. [7] introduced an effective tree energy loss based on the low-level and high-level features for point/scribble/blocksupervised semantic segmentation. For semantic segmentation, the supervision of image-level labels has been well explored [12, 37, 15, 14]. Recently, some works have been proposed to make use of the large-scale pretrained CLIP model [38] to achieve weakly-supervised or annotation-free semantic

Network

Pairwise Affinity Propagation

Global affinity propagation

Local affinity propagation

+

(a)

(b)

(

)

f

I



Y

Y

I

g

u

segmentation [39, 40]. In addition, weakly-supervised panoptic segmentation methods [3, 41, 4] with a single and multiple points annotation have been proposed. In this paper, we aim to develop a universal component for various segmentation tasks, which can be easily plugged into the existing segmentation networks.

Pairwise Affinity Modeling. Pairwise affinity modeling plays an important role in many computer vision tasks. Classical methods, like CRF [42], make full use of the color and spatial information to model the pairwise relations in the semantic labeling space. Some works use it as a post-processing module [43], while others integrate it as a jointly-trained part into the deep neural network [44, 45]. Inspired by CRF, some recent approaches explore the local appearance kernel to tap the neighboring pairwise affinities, where spatially nearby pixels with similar colors are more likely to fall into the same class. Tian et al. [8] proposed the local pairwise loss, which models the neighboring relationship in LAB color space. Some works [10, 16, 17] focus on the pixel relation in RGB color space directly, and achieve competitive performance. However, the local operation fails to capture global context cues and lacks the long-range affinity dependencies. Furthermore, the appearance kernel cannot reflect the topology of objects, missing the details of semantic objects. To model the structural pair-wise relationship, tree-based approaches [7, 19] leverage the minimum spanning tree [20] to capture the topology of objects in the image. However, the affinity interactions with distant nodes decay rapidly as the distance increases along the spanning tree. Besides, Zhang et al. [35] proposed an affinity network to convert an image to a weighted graph, and model the node affinities by the graph neural network (GNN). Different from these methods, in this work we model the pairwise affinity from a new affinity propagation perspective both globally and locally.

3 Methodology

In this section, we introduce our proposed affinity propagation ( APro ) approach to label-efficient segmentation. First, we define the problem and model the APro framework in Section 3.1. Then, we describe the detailed pairwise affinity propagation method in Section 3.2. Finally, we provide an efficient implementation of our method in Section 3.3.

3.1 Problem Definition and Modeling

Given an input image I = { x i } N with N pixels and its available sparse ground truth labeling ¯ Y ( i.e. points, scribble, or bounding box), and let f θ ( I ) be the output of a segmentation network with the learnable parameters θ , the whole segmentation network can be regarded as a neural network optimization problem as follows:

where L g is a ground truth loss on the set of labeled pixels Ω g with ¯ Y and L u is a loss on the unlabeled regions Ω u with the pseudo label Y . As shown in Fig. 2-(a), our goal is to obtain the accurate pseudo label Y by leveraging the image pixel affinities for unlabeled regions.

As in the classic MRF/CRF model [46, 42], the unary term reflects the per-pixel confidence of assigning labels, while pairwise term captures the inter-pixel constraints. We define the generation of

pseudo label Y as an affinity propagation process, which can be formulated as follows:

where ϕ ( x j ) denotes the unary potential term, which is used to align y j and the corresponding network prediction based on the available sparse labels. ψ ( x i , x j ) indicates the pairwise potential, which models the inter-pixel relationships to constrain the predictions and produce accurate pseudo labels. τ is the region with different receptive fields. z i is the summation of pairwise affinity ψ ( x i , x j ) along with j to normalize the response.

Notably, we unify both global and local pairwise potentials in an affinity propagation process formulated in Eq. 2. As shown in Fig. 2-(b), the global affinity propagation (GP) can capture the pairwise affinities with topological consistency in a global view, while the local affinity propagation (LP) can obtain the pairwise affinities with local spatial consistency. Through the proposed component, the soft pseudo labels Y can be obtained. We assign each y i from GP and LP with the network prediction p i and directly employ the distance measurement function as the objective for unlabeled regions Ω u . Simple L 1 distance is empirically adopted in our implementation.

3.2 Pairwise Affinity Propagation

3.2.1 Global Affinity Propagation

We firstly provide a solution to model the global affinity efficiently based on the input image. Specifically, we represent an input image as a 4-connected planar graph G , where each node is adjacent to up to 4 neighbors. The weight of the edge measures the image pixel distance between adjacent nodes. Inspired by tree-based approaches [47, 19], we employ the minimum spanning tree (MST) algorithm [20] to remove the edge with a large distance to obtain the tree-based sparse graph G T , i.e. , G T ← MST ( G ) and G T = {V , E} , where V = {V i } N is the set of nodes and E = {E i } N -1 denotes the set of edges.

Then, we model the global pairwise potential by iterating over each node. To be specific, we take the current node as the root of the spanning tree G T and propagate the long-range affinities to other nodes. While the distant nodes along the spanning tree need to pass through nearby nodes along the path of spanning tree, the distance-insensitive max affinity function can alleviate this geometric constraint and relax the affinity decay for long-range nodes. Hence, we define the global pairwise potential ψ g as follows:

where T denotes the global tree. E i,j is the set of edges in the path of T from node j to node i . w k,l indicates the edge weight between the adjacent nodes k and l , which is represented as the Euclidean distance between pixel values of two adjacent nodes, i.e. , w k,l = | I k -I l | 2 . ζ g controls the degree of similarity with the long-range pixels. In this way, the global affinity propagation (GP) process to obtain the soft label y g can be formulated as follows:

where interactions with distant nodes are performed over tree-based topology. In addition to utilizing low-level image, we empirically employ high-level feature as input to propagate semantic affinity.

3.2.2 Local Affinity Propagation

The long-range pairwise affinity is inevitably noisy since it is computed based on the susceptible image features in a global view. The spatially nearby pixels are more likely to have the same label, while they have certain difference in color and intensity, etc . Hence, we further introduce the local affinity propagation (LP) to promote the piece-wise smoothness. The Gaussian kernel is widely used to capture the local relationship among the neighbouring pixels in previous works [45, 8, 10]. Different from these works, we define the local pairwise affinity via the formulated affinity propagation process. The local pairwise term ψ s is defined as:

where K denotes the Gaussian kernel, N ( i ) is the set containing all local neighbor pixels. The degree of similarity is controlled by parameter ζ s . Then the pseudo label y s can be obtained via the following affinity propagation:

where the local spatial consistency is maintained based on high-contrast neighbors. To obtain a robust segmentation performance, multiple iterations are required. Notably, our LP process ensures a fast convergence, which is 5 × faster than MeanField-based method [10, 42]. The details can be found in the experimental section 4.4.

3.3 Efficient Implementation

Given a tree-based graph G T = {V , E} in the GP process, we define the maximum w value of the path through any two vertices as the transmission cost C . One straightforward approach to get y g i of vertex i is to traverse each vertex j by Depth First Search or Breadth First Search to get the transmission cost C i,j accordingly. Consequently, the computational complexity required to acquire the entire output y g is O ( N 2 ) , making it prohibitive in real-world applications.

Instead of calculating and updating the transmission cost of any two vertices, we design a lazy update algorithm to accelerate the GP process. Initially, each node is treated as a distinct union, represented by U . Unions are subsequently connected based on each edge w k,l in ascending order of w . We show that when connecting two unions U k and U l , w k,l is equivalent to the transmission cost for all nodes within U k and U l . This is proved in the supplementary material .

To efficiently update values, we introduce a Lazy Propagation scheme. We only update the value of the root node and postpone the update of its descendants. The update information is retained in a lazy tag Z and is updated as follows:

where S ( δ ) i = ∑ j ∈U i δ j , δ means different inputs, including the dense prediction ϕ ( x ) and all-one matrix Λ . k ∗ /l ∗ denotes the root node of node k/l .

Once all unions are connected, the lazy tags can be propagated downward from the root node to its descendants. For the descendants, the global affinity propagation term is presented as follows:

where Asc G T ( i ) represents the ascendants of node i in the tree G T . As shown in Algorithm 1, the disjoint-set data structure is employed to implement the proposed algorithm. In our implementation, a Path Compression strategy is applied, connecting each node on the path directly to the root node. Consequently, it is sufficient to consider the node itself and its parent node to obtain LProp .

Time complexity. For each channel of the input, the average time complexity of sorting is O ( N log N ) . In the merge step, we utilize the Path Compression and Union-by-Rank strategies, which have a complexity of O ( α ( N )) [48]. After merging all the concatenated blocks, the lazy tags can be propagated in O ( N ) time. Hence, the overall complexity is O ( N log N ) . Note that the batches and channels are independent of each other. Thus, the algorithm can be executed in parallel for both batches and channels for practical implementations. As a result, the proposed algorithm reduces the computational complexity dramatically.

4 Experiments

4.1 Weakly-supervised Instance Segmentation

Datasets. As in prior arts [8, 9, 16, 17], we conduct experiments on two widely used datasets for the weakly box-supervised instance segmentation task:

COCO [49], which has 80 classes with 115K train2017 images and 5K val2017 images.

Algorithm 1: Algorithm for GP process

Pascal VOC [43] augmented by SBD [50] based on the original Pascal VOC 2012 [51], which has 20 classes with 10,582 trainaug images and 1,449 val images.

Base Architectures and Competing Methods. In the evaluation, we apply our proposed APro to two representative instance segmentation architectures, SOLOv2 [52] and Mask2Former [53], with different backbones ( i.e. , ResNet [54], Swin-Transformer [55]) following Box2Mask [11]. We compare our approach with its counterparts that model the pairwise affinity based on the image pixels without modifying the base segmentation network for box-supervised setting. Specifically, the compared methods include Pairwise Loss [8], TreeEnergy Loss [7] and CRF Loss [10]. For fairness, we re-implement these models using the default setting in MMDetection [56].

Implementation Details. We follow the commonly used training settings on each dataset as in MMDetection [56]. All models are initialized with ImageNet [57] pretrained backbone. For SOLOv2 framework [52], the scale jitter is used, where the shorter image side is randomly sampled from 640 to 800 pixels. For Mask2Former framework [53], the large-scale jittering augmentation scheme [58] is employed with a random scale sampled within range [0.1, 2.0], followed by a fixed size crop to 1024×1024. The initial learning rate is set to 10 -4 and the weight decay is 0.05 with 16 images per mini-batch. The box projection loss [8, 9] is employed to constrain the network prediction within the bounding box label as the unary term ϕ . COCO-style mask AP (%) is adopted for evaluation.

Quantitative Results. Table 1 shows the quantitative results. We compare the approaches with the same architecture for fair comparison. The state-of-the-art methods are listed for reference. One can see that our APro method outperforms its counterparts across Pascal VOC and COCO datasets.

Pascal VOC [43] val . Under the SOLOv2 framework, our approach achieves 37.1% AP and 38.4% AP with 12 epochs and 36 epochs, respectively, outperforming other methods by 1.4%-2.5% mask AP with ResNet-50. With the Mask2Former framework, our approach also outperforms its counterparts. Furthermore, with the Swin-L backbone [55], our proposed approach achieves very promising performance, 49.6% mask AP with 50 epochs.

COCO [49] val . Under the SOLOv2 framework, our approach achieves 32.0% AP and 32.9% AP with 12 epochs and 36 epochs, and surpasses its best counterpart by 1.0% AP and 0.4% AP using ResNet-50, respectively. Under the Mask2Former framework, our method still achieves the best performance with ResNet-50 backbone. Furthermore, equipped with stronger backbones, our approach obtains more robust performance, achieving 38.0% mask AP with ResNet-101, and 41.0% mask AP using Swin-L backbone.

Qualitative Results. Fig. 3 illustrates the visual comparisons on affinity maps of our APro and other approaches, and Fig. 4 compares the segmentation results. One can clearly see that our method captures accurate pairwise affinity with object's topology and yields more fine-grained predictions.

4.2 Weakly-supervised Semantic Segmentation

Datasets. We conduct experiments on the widely-used Pascal VOC2012 dataset [51], which contains 20 object categories and a background class. As in [6, 7], the augmented Pascal VOC dataset is adopted here. The point [2] and scribble [5] annotations are employed for weakly point-supervised and scribble-supervised settings , respectively.

Implementation Details. As in [7], we adopt LTF [19] as the base segmentation model. The input size is 512 × 512. The SGD optimizer with momentum of 0.9 and weight decay of 10 -4 is used. The initial learning rate is 0.001, and there are 80k training iterations. The same data augmentations as in [7] are utilized. We employ the partial cross-entropy loss to make full use of the available

† :adopting multi-stage training, ∗ :our re-implementation.

point/scribble labels and constrain the unary term. ResNet-101 [54] pretrained on ImageNet [57] is adopted as backbone network for all methods.

Quantitative Results. As shown in Table 2, we compare our APro approach with the state-of-the-art methods on point-supervised and scribble-supervised semantic segmentation, respectively.

Point-wise supervision. With DeepLabV2 [43], KernelCut Loss [6] achieves 57.0% mIoU. Equipped with LTF [19], TEL [7] achieves 66.8% mIoU. Our APro achieves 67.7% mIoU, outperforming the previous best method TEL [7] by 0.9% mIoU.

Scribble-wise supervision. The scribble-supervised approaches are popular in weakly supervised semantic segmentation. We apply the proposed approach under the single-stage training framework without calling for CRF post-processing during testing. Compared with the state-of-the-art methods, our approach achieves better performance with 76.6% mIoU.

Image

GP

TF

LP

Init.

Soft pseudo

L#

CHEETO

4.3 CLIP-guided Semantic Segmentation

Datasets. To more comprehensively evaluate our proposed approach, we conduct experiments on CLIP-guided annotation-free semantic segmentation with three widely used datasets:

Pascal VOC 2012 [51] introduced in Section 4.2.

Pascal Context [60], which contains 59 foreground classes and a background class with 4,996 train images and 5,104 val images.

COCO-Stuff [61], which has 171 common semantic object/stuff classes on 164K images, containing 118,287 train images and 5,000 val images.

Base Architectures and Backbones. We employ MaskCLIP+ [40] as our base architecture, which leverages the semantic priors of pretrained CLIP [38] model to achieve the annotation-free dense semantic segmentation. In the experiments, we couple MaskCLIP+ with our APro approach under ResNet-50, ResNet-50 × 16 and ViT-B/16 [62]. The dense semantic predictions of MaskCLIP [40] are used as the unary term, and our proposed method can refine it and generate more accurate pseudo labels for training target networks.

Implementation Details. For fair comparison, we keep the same settings as MaskCLIP+ [40]. We keep the text encoder of CLIP unchanged and take prompts with target classes as the input. For text embedding, we feed prompt engineered texts into the text encoder of CLIP with 85 prompt templates, and average the results with the same class. For ViT-B/16, the bicubic interpolation is adopted for the pretrained positional embeddings. The initial learning rate is set to 10 -4 . We train all models with batch size 32 and 2k/4k/8k iterations. DeepLabv2-ResNet101 is used as the backbone.

Quantitative Results. Table 3 compares our approach with MaskCLIP+ [40] for annotaion-free semantic segmentation. We have the following observations.

Pascal VOC2012 [51] val . With ResNet-50 as the image encoder in pretrained CLIP model, our approach

outperforms MaskCLIP+ by 3.6% mIoU. With ResNet-50 × 16 and ViT-B/16 as the image encoders, our method surpasses MaskCLIP+ by 2.9% and 1.5% mIoU, respectively.

Pascal Context [60] val . Our proposed method outperforms MaskCLIP+ consistently with different image encoders (about +1.5% mIoU).

COCO-Stuff [61] val . COCO-Stuff consists of hundreds of semantic categories. Our method still brings +1.0%, +0.9% and +1.5% performance gains over MaskCLIP+ with ResNet-50, ResNet50 × 16 and ViT-B/16 image encoders, respectively.

only GP

only LP

APro

INJECTABLE GEL

4.4 Diagnostic Experiments

For in-depth analysis, we conduct ablation studies on Pascal VOC [51] upon the weakly boxsupervised instance segmentation task.

Unary and Pairwise Terms. Table 4 shows the evaluation results with different unary and pairwise terms. When using the unary term only, our method achieves 25.9% AP. When the global pairwise term is employed, our method achieves a much better performance

of 36.3% AP. Using the local pairwise term only, our method obtains 36.0% AP. When both the global and local pairwise terms are adopted, our method achieves the best performance of 38.4% AP.

Tree-based Long-range Affinity Modeling. The previous works [19, 7] explore treebased filters for pairwise relationship modeling. Table 5 compares our method with them. TreeFilter can capture the relationship with dis-

tant nodes to a certain extent (see Fig. 3). Directly using TreeFilter as the pairwise term leads to 36.1% AP. By combining TreeFilter with our local pairwise term, the model obtains 36.8%AP. In comparison, our proposed approach achieves 38.4% AP.

Iterated Local Affinity Modeling. We evaluate our local affinity propagation (LP) with different iterations, and compare it with the classical MeanField method [10, 42]. Table 6 reports the comparison results. Our APro with the LP process achieves 36.0% AP after 20 iterations. However, replacing our local affinity propagation with MeanFiled-based method [10] costs 100 iterations to obtain 35.9% AP. This indicates that our LP method possesses the attribute of fast convergence.

Soft Pseudo-label Generation. With the formulated GP and LP methods, we study how to integrate them to generate the soft pseudolabels in Table 7. We can cascade GP and LP sequentially to refine the pseudo labels. Putting GP before LP (denoted as GP-LP-C) achieves 36.8% AP, and putting LP before GP (denoted as LPGP-C) performs better with 37.7% AP. In addition, we can use

GP and LP in parallel (denoted as GP-LP-P) to produce two pseudo labels, and employ both of them to optimize the segmentation network with L 1 distance. Notably, GP-LP-P achieves the best performance with 38.4% mask AP. This indicates that our proposed affinity propagation in global and local views are complementary for optimizing the segmentation network.

Runtime Analysis. We report the average runtime of our method in Table 8. The experiment is conducted on a single GeForce RTX 3090 with batch size 1. Here we report the average runtime for one GP process duration of an epoch on the Pascal VOC dataset. When directly using Breadth First Search for each node with N times, the runtime is 4.3 × 10 3 ms with O ( N 2 ) time complexity. While employing the

proposed efficient implementation, the runtime is only 0.8 ms with O ( N log N ) time complexity. This demonstrates that the proposed efficient implementation reduces the computational complexity dramatically.

5 Conclusion

In this work, we proposed a novel universal component for weakly-supervised segmentation by formulating it as an affinity propagation process. A global and a local pairwise affinity term were introduced to generate the accurate soft pseudo labels. An efficient implementation with the light computational overhead was developed. The proposed approach, termed as APro , can be embedded into the existing segmentation networks for label-efficient segmentation. Experiments on three typical label-efficient segmentation tasks, i.e. , box-supervised instance segmentation, point/scribblesupervised semantic segmentation and CLIP-guided annotation-free semantic segmentation, proved the effectiveness of proposed method.

Acknowledgments

This work is supported by National Natural Science Foundation of China under Grants (61831015). It is also supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

References

Wei Shen, Zelin Peng, Xuehui Wang, Huayu Wang, Jiazhong Cen, Dongsheng Jiang, Lingxi Xie, Xiaokang Yang, and Q Tian. A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction. TPAMI , 2023. 1, 2

Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In ECCV , pages 549-565, 2016. 1, 2, 7

Junsong Fan, Zhaoxiang Zhang, and Tieniu Tan. Pointly-supervised panoptic segmentation. In ECCV , pages 319-336, 2022. 1, 3

Wentong Li, Yuqian Yuan, Song Wang, Jianke Zhu, Jianshu Li, Jian Liu, and Lei Zhang. Point2mask: Point-supervised panoptic segmentation via optimal transport. In ICCV , pages 572-581, 2023. 1, 3

Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR , pages 3159-3167, 2016. 1, 2, 7

Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. On regularized losses for weakly-supervised cnn segmentation. In ECCV , pages 507-522, 2018. 2, 7

Zhiyuan Liang, Tiancai Wang, Xiangyu Zhang, Jian Sun, and Jianbing Shen. Tree energy loss: Towards sparsely annotated semantic segmentation. In CVPR , pages 16907-16916, 2022. 1, 2, 3, 6, 7, 9

Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: High-performance instance segmentation with box annotations. In CVPR , pages 5443-5452, 2021. 1, 2, 3, 4, 5, 6, 7

Wentong Li, Wenyu Liu, Jianke Zhu, Miaomiao Cui, Xian-Sheng Hua, and Lei Zhang. Box-supervised instance segmentation with level set evolution. In ECCV , pages 1-18, 2022. 2, 5, 6, 7

Shiyi Lan, Xitong Yang, Zhiding Yu, Zuxuan Wu, Jose M Alvarez, and Anima Anandkumar. Vision transformers are good mask auto-labelers. In CVPR , 2023. 1, 2, 3, 4, 5, 6, 7, 9

Wentong Li, Wenyu Liu, Jianke Zhu, Miaomiao Cui, Risheng Yu, Xiansheng Hua, and Lei Zhang. Box2mask: Box-supervised instance segmentation via level-set evolution. arXiv preprint arXiv:2212.01579 , 2022. 1, 2, 6

Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR , pages 4981-4990, 2018. 1, 2

Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In CVPR , pages 2209-2218, 2019. 1

Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du. Learning affinity from attention: end-to-end weaklysupervised semantic segmentation with transformers. In CVPR , pages 16846-16855, 2022. 1, 2

Jinlong Li, Zequn Jie, Xu Wang, Xiaolin Wei, and Lin Ma. Expansion and shrinkage of localization for weakly-supervised semantic segmentation. In NeurIPS , 2022. 1, 2

Cheng-Chun Hsu, Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, and Yung-Yu Chuang. Weakly supervised instance segmentation using the bounding box tightness prior. In NeurIPS , volume 32, 2019. 1, 2, 3, 5, 7

Shiyi Lan, Zhiding Yu, Christopher Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry S Davis, and Anima Anandkumar. Discobox: Weakly supervised instance segmentation and semantic correspondence from box supervision. In ICCV , pages 3406-3416, 2021. 1, 2, 3, 5, 7

Paul Vernaza and Manmohan Chandraker. Learning random-walk label propagation for weakly-supervised semantic segmentation. In CVPR , pages 7158-7166, 2017. 1

Lin Song, Yanwei Li, Zeming Li, Gang Yu, Hongbin Sun, Jian Sun, and Nanning Zheng. Learnable tree filter for structure-preserving feature transform. In NeurIPS , volume 32, 2019. 2, 3, 4, 7, 8, 9

Joseph B Kruskal. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical society , 7(1):48-50, 1956. 2, 3, 4

Lin Song, Yanwei Li, Zhengkai Jiang, Zeming Li, Xiangyu Zhang, Hongbin Sun, Jian Sun, and Nanning Zheng. Rethinking learnable tree filter for generic feature transform. In NeurIPS , volume 33, pages 3991-4002, 2020. 2

Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In ICML , pages 912-919, 2003. 2

Xiaojin Zhu. Semi-supervised learning literature survey. 2005. 2

Alexander Vezhnevets and Joachim M Buhmann. Towards weakly supervised semantic segmentation by means of multiple instance and multitask learning. In CVPR , pages 3249-3256, 2010. 2

Deepak Pathak, Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional multi-class multiple instance learning. arXiv preprint arXiv:1412.7144 , 2014.

Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. In ICCV , pages 1796-1804, 2015.

Pedro O Pinheiro and Ronan Collobert. From image-level to pixel-level labeling with convolutional networks. In CVPR , pages 1713-1721, 2015.

George Papandreou, Liang-Chieh Chen, Kevin P Murphy, and Alan L Yuille. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In ICCV , pages 1742-1750, 2015. 2

Ruihuang Li, Chenhang He, Yabin Zhang, Shuai Li, Liyi Chen, and Lei Zhang. Sim: Semantic-aware instance mask generation for box-supervised instance segmentation. In CVPR , 2023. 2

Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Qian Zhang, and Wenyu Liu. Boxteacher: Exploring high-quality pseudo labels for weakly supervised instance segmentation. In CVPR , 2023. 2

Rui Yang, Lin Song, Yixiao Ge, and Xiu Li. Boxsnake: Polygonal instance segmentation with box supervision. In ICCV , 2023. 2

Bowen Cheng, Omkar Parkhi, and Alexander Kirillov. Pointly-supervised instance segmentation. In CVPR , pages 2617-2626, 2022. 2

Chufeng Tang, Lingxi Xie, Gang Zhang, Xiaopeng Zhang, Qi Tian, and Xiaolin Hu. Active pointlysupervised instance segmentation. In ECCV , pages 606-623, 2022. 2

Hongjun Chen, Jinbao Wang, Hong Cai Chen, Xiantong Zhen, Feng Zheng, Rongrong Ji, and Ling Shao. Seminar learning for click-level weakly supervised semantic segmentation. In ICCV , pages 6920-6929, 2021. 2

Bingfeng Zhang, Jimin Xiao, Jianbo Jiao, Yunchao Wei, and Yao Zhao. Affinity attention graph neural network for weakly supervised semantic segmentation. TPAMI , 44(11):8082-8096, 2021. 2, 3

Dmitrii Marin, Meng Tang, Ismail Ben Ayed, and Yuri Boykov. Beyond gradient descent for regularized segmentation losses. In CVPR , pages 10187-10196, 2019. 2, 7

Yun Liu, Yu-Huan Wu, Peisong Wen, Yujun Shi, Yu Qiu, and Ming-Ming Cheng. Leveraging instance-, image-and dataset-level information for weakly supervised instance segmentation. TPAMI , 44(3):14151428, 2020. 2

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021. 2, 8

Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In CVPR , 2023. 3

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV , pages 696-712, 2022. 3, 8

Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Yukang Chen, Lu Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation with point-based supervision. TPAMI , 2022. 3

Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NeurIPS , volume 24, 2011. 3, 5, 9

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI , 40(4):834-848, 2017. 3, 6, 7

Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. Conditional random fields as recurrent neural networks. In ICCV , pages 1529-1537, 2015. 3

Anton Obukhov, Stamatios Georgoulis, Dengxin Dai, and Luc Van Gool. Gated crf loss for weakly supervised semantic image segmentation. In NeurIPS , 2019. 3, 4

William T Freeman, Egon C Pasztor, and Owen T Carmichael. Learning low-level vision. IJCV , 40:25-47, 2000. 3

Qingxiong Yang. Stereo matching using tree filtering. IEEE TPAMI , 37(4):834-846, 2014. 4

Robert Endre Tarjan. A class of algorithms which require nonlinear time to maintain disjoint sets. Journal of computer and system sciences , 18(2):110-127, 1979. 5

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755, 2014. 5, 6, 7

Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV , pages 991-998, 2011. 6

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 88(2):303-338, 2010. 6, 7, 8, 9

Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. In NeurIPS , volume 33, pages 17721-17732, 2020. 6

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In CVPR , pages 1290-1299, 2022. 6

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016. 6, 7

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , pages 10012-10022, 2021. 6

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019. 6

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet: large scale visual recognition challenge. IJCV , 115:211-252, 2015. 6, 7

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR , pages 2918-2928, 2021. 6

Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers. Normalized cut loss for weakly-supervised cnn segmentation. In CVPR , pages 1818-1827, 2018. 7

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR , pages 891-898, 2014. 8

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR , 2018. 8

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2020. 8

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel universal component for weakly-supervised segmentation by formulating it as an affinity propagation process. It simultaneously utilizes a global and a local pairwise affinity term to generate soft pseudo labels. An efficient algorithm is also developed to reduce the computational cost. Experiments on three label-efficient segmentation tasks demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The proposed framework uses both global and local pairwise affinity term and achieves superior performance.\n2. The efficient implementation of global affinity propagation can greatly reduce computational cost.\n3. The proposed approach can be conveniently plugged into existing segmentation networks.\n4. The experiments are abundant, covering many label-efficient segmentation tasks.\n\nWEAKNESSES:\n1. This approach seems parameter-sensitive. The slight variation of $\\zeta_s$, $\\zeta_g$ may lead to notable fluctuation of segmentation performance. Is there consistency in the parameters used across different tasks and datasets?\n2. The efficiency of the whole framework is not intuitive. By introducing both global and local pairwise affinity term, is there a significant decrease in efficiency? It would be helpful if authors could present the change of training time with/without APro.\n\nQUESTIONS:\n1. How to determine the number of iterations?\n2. The runtime details in Table 7 are not clear, such as iterations, batch size, device...\n3. In MaskCLIP, key smoothing and prompt denoising are proposed to refine the pseudo masks. The key smoothing also aims to realize global affinity propagation based on the similarity of key features (not used in the MaskCLIP+ setting). It would be helpful to explore their effects.\n\nLIMITATIONS:\nPlease refer to the Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an affinity propagation algorithm for weakly supervised segmentation. Given image segmentation data with only sparse box/point/scribbles annotations, the proposed method propagates these annotations to other pixels as pseudo masks for training. Global affinity and local affinity are proposed to capture global pairwise potential and local connectivity respectively. The authors propose an efficient implementation for the propagation of these two types of affinity which makes training practical. The final model performs better than prior works in PASCAL VOC and COCO with partial annotations.\n\nSTRENGTHS:\n- The proposed framework is simple and effective. Both global and local affinity propagation are intuitive, and the resultant method achieves strong performance compared to existing works without bells and whistles. This shows the effectiveness of the proposed affinity propagation mechanism.\n\n- The authors propose an efficient implementation for the affinity propagation algorithm which is crucial for making training with this method practical. The authors claim that it is five times faster than mean-field [29]. This fast algorithm is also potentially useful for future work in other directions.\n\nWEAKNESSES:\n- The affinities are defined on pixel intensity which can be limiting as it is sensitive to lighting and low-level noises. It might help to incorporate deep features from self-supervised, pre-trained networks (e.g., MAE).\n\n- Another direction for segmentation is open-vocabulary segmentation which can benefit from scaling and data engines (e.g., SAM). The proposed method does not seem to be scalable as the affinity is fixed to low-level color differences.\n\n- Missing related work and discussion: Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018\n\nQUESTIONS:\n- Will the code be released? I see no mention of this in the paper. It would help the community if an open-source version is available, especially for the non-trivial GPU implementation of affinity propagation.\n\n- When comparing speed with mean-field [29], are both algorithms running on GPU?\n\nLIMITATIONS:\nLimitations are mentioned in Section D in the supplementary material. The authors mention that the use of only image intensity to compute affinity is a limitation which I agree with.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a weakly-supervised segmentation framework based on affinity propagation. It overcomes the drawback of simply modeling neighboring pairwise potentials and proposes both global and local affinity terms to generate pseudo labels. The authors demonstrate the effectiveness of the proposed method in box-supervised instance segmentation, point/scribble-supervised semantic segmentation, and CLIP-guided semantic segmentation tasks.\n\nSTRENGTHS:\nOriginality: The paper proposes two kinds of pairwise affinity propagation which are novel for weakly-supervised image segmentation settings. The global affinity propagation employs a minimum spanning tree to remove the edge with a large distance to obtain the tree-based sparse graph, which explicitly captures long-range dependency, while local affinity propagation leverages the conventional Gaussian kernel.  The combination of the two enhances the performances based on Table 4. \n\nQuality: The paper itself is self-contained and includes sufficient experimental setup and details. \n\nClarity: The paper is well-written but it might be hard to follow. It would be hard to understand the context of the proposed method for my initial reading and still creates some confusion after repeated reading. Figures are suggested to enhance the clarity of certain concepts developed in this paper. \n\nSignificance: I give high significance to this paper, as it proposes an effective and convincing method for a well-known and challenging problem.\n\nWEAKNESSES:\nPersonally, the biggest concern of this paper is the clarification. Basically, the authors devised two affinity scheme to create long-range and short-distance dependencies and both of them uses the same framework as described in Equ. 2. However, it would be very hard to understand the details of the global one. Authors are suggested to make a graphical illustration to demonstrate the idea.  Also, it still remains confusion about the method, as will be asked in the next section.\n\nQUESTIONS:\nIt would be appreciated if the authors can kindly reply to my questions and concerns. I will consider raising my evaluation. \n\n- The whole framework of affination is defined on the classic CRF model, as cited in line 110. However, it is noted that in [33], the energy function is the sum of unaries and the sum of pairwise potentials. The CRF model in Equ. 2 turns out to be the sum of products between unary and pairwise terms. If understand correctly, this formulation can be interpreted as the weighted sum of unaries defined on neighbor pixels, with weights calculated from pairwise terms. I am wondering how the authors draw an equivalence between the two formulations. \n\n- What is the $\\mathcal{L}_{g}$ used in Equ. 1? is it a partial cross-entropy? how is it defined with non-annotated pixels? The unary used in Equ. 2 is then the output distribution of only labeled pixels or all pixels?\n\n- In 3.2.1, the node in line 127 is defined as a pixel right? why the set of edges can have N-1 elements, as defined in line 132\n\n- How do authors set the degree of similarity defined on line 143? is it unique for all nodes?\n\n- Can the authors explain why the tree-based model preserves topology?\n\n- For the local affinity propagation, I always consider Equ. 6 as a sum of neighbor pixels weighted by the intensities, similar to the conventional CRF model [33]. However, in [33], a spatial smoothness term is added. The authors want to explain how their method preserves spatial smoothness. \n\n- Based on Equ.6  I don't see any necessity of iterating it (there are even no iterative parameters inside). What does it mean by saying iterating local affinity leads to better performance? \n\n- what is the transmission cost? \n\n\n- In 3.2.1 and 3.2.1, the authors then obtain two sets of pseudo labels, $y^{g}$ and $y^{s}$. How do they exactly train the network with these two sets of labels? How to merge them when using them together?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an affinity propagation method within local and global perspectives to improve the pseudo labels generated by the model for the parts without GT masks. Meanwhile, they then propose an efficient implementation to solve the heavy computation by graph modeling.  The authors conduct experiments on several benchmarks to showcase their improvements on point/scribble/bbox-level weakly-supervised semantic segmentation, acting as a plug-in module to enhance their segmentation performance. And competitive performances are obtained naturally.\n\nSTRENGTHS:\n1. The writing of this paper is easy to follow.\n2. Extensive experiments are studied and analyzed to present the proposed method.\n3. The final performances are competitive.\n\nWEAKNESSES:\n1. To my concern, affinity propagation exploration has been studied widely in the weakly-supervised learning community, such as [1]. The overall novelty of this paper sounds limited because their method has a lot of overlappings with previous method, though the author adapt a graph modeling to analyze and model the propagation procedure. It is important to demonstrate the main differences from the proposed method with others.\n\n    [1] Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers, CVPR2022\n\n2. From the eq3 and eq5, it is hard to get the point, of how they define the global and local respectively since the main differences lie in that a max operation is performed in eq3 and the points-set in eq5 is local but not detailed clearly.\n\n3. The motivation for adapting Gaussian to model the local part in this paper should present more reasonably, for example, by giving some experiments directly to see that the Gaussian methods indeed capture the local receptive features.\n\n4. Despite the point/scribble/bbox-level weakly supervised semantic segmentation, how about the results when performing on semi-/image-level supervised semantic segmentation tasks to validate that the proposed method indeed helps to improve the soft-pseudo label for the unlabeled pixels?\n\n5. Some important references are missing below while these methods may discuss in the paper too.\n\n    [2] Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation, NeurIPS 2021\n\n    [3] Expansion and Shrinkage of Localization for Weakly-Supervised Semantic Segmentation, NeurIPS 2022\n\n    [4] Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation, CVPR2023\n\n    [5] Semi-Supervised Semantic Segmentation With Error Localization Network, CVPR2022\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nDespite the limited novelty in this phase, this paper can be stronger when the authors well address my questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilizes local and global pairwise affinity terms to generate accurate soft pseudo labels and incorporates an efficient algorithm to reduce computational costs. Experimental results demonstrate the approach's superior performance in various segmentation tasks.\n\nSTRENGTHS:\nExperimental results demonstrate the approach's superior performance in various segmentation tasks.\nThe paper is easy to understand.\n\nWEAKNESSES:\nThe affinity methods are commonly used in weakly supervised segmentation tasks, such as 'Learning Affinity from Attention: End-to-End Weakly-Supervised SemanticSegmentation with Transformers\nAffinity Attention Graph Neural Network for Weakly Supervised Semantic Segmentation.\nLearning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation\nWeakly Supervised Learning of Instance Segmentation with Inter-pixel Relations.'\n\nThe novelty of their method might be limited compared to existing affinity-based approaches. To stand out, the author should clearly highlight the differences between their method and the mentioned ones. It could be in terms of the formulation of the affinity modeling task, the incorporation of local and global pairwise affinity terms, the generation of accurate soft pseudo labels, or the development of an efficient algorithm to reduce computational costs.\n\nProviding a thorough comparison with these existing methods would help readers understand the unique contributions of the proposed approach and its advantages over previous approaches. By highlighting these differences, the paper can demonstrate why their method is valuable and relevant in the context of weakly supervised segmentation tasks.\n\nQUESTIONS:\nTo make their contribution more apparent, the author should provide a comprehensive comparison of the proposed method's performance with previous approaches of the generated pseudo ground truth. This comparison would highlight the advantages and improvements of their approach over existing methods in terms of accuracy, efficiency, and other relevant metrics.\n\nBy including a thorough analysis of the pseudo ground truth performance, readers can better understand the strengths of the proposed approach and how it outperforms or complements existing techniques.\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}