P2Seg: Pointly-supervised Segmentation via Mutual Distillation
Authors: Zipeng Wang
Xuehui Yu
Xumeng Han
Wenwen Yu
Zhixun Huang
Jianbin Jiao
Zhenjun Han
Venue: ICLR 2024 poster (2024)
Keywords: Mutual Distillation
Semantic to Instance
Instance to Semantic
Point-level Supervised Instance Segmentation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: By employing mutual distillation, we achieve adjacent object boundary prediction and address the local response caused by the separation of points and image features.

Abstract: Point-level Supervised Instance Segmentation (PSIS) aims to enhance the applicability and scalability of instance segmentation by utilizing low-cost yet instance-informative annotations. Existing PSIS methods usually rely on positional information to distinguish objects, but predicting precise boundaries remains challenging due to the lack of contour annotations. Nevertheless, weakly supervised semantic segmentation methods are proficient in utilizing intra-class feature consistency to capture the boundary contours of the same semantic regions. In this paper, we design a Mutual Distillation Module (MDM) to leverage the complementary strengths of both instance position and semantic information and achieve accurate instance-level object perception. The MDM consists of Semantic to Instance (S2I) and Istance to Semantic (I2S). S2I is guided by the precise boundaries of semantic regions to learn the association between annotated points and instance contours. I2S leverages discriminative relationships between instances to facilitate the differentiation of various objects within the semantic map. Extensive experiments substantiate the efficacy of MDM in fostering the synergy between instance and semantic information, consequently improving the quality of instance-level object representations. Our method achieves 55.7 mAP50 and 17.6 mAP on the PASCAL VOC and MS COCO datasets, significantly outperforming recent PSIS methods and several box-supervised instance segmentation competitors.

Published as a conference paper at ICLR 2024
P2SEG: POINTLY-SUPERVISED SEGMENTATION VIA MUTUAL DISTILLATION
Zipeng Wang' , Xuehui Yu Jianbin Jiao', Zhenjun Han Universiity of Chinese Academy of Sciences Sangzipeng222mai11 ucas acc.ca
Xumeng Han', Wenwen Yu', Zhixun Huang
Xiaaoii Al Lab, Beijing, China
ABSTRACT
Point-level Supervised Instance Segmentation (PSIS) aims enhance the applica- bility and scalability of instance segmentation by utilizing low-cost yet instance- informative annotations, Existing PSIS methods usually rely on positional infor- mation to distinguish objects, but predicting precise boundaries remains challeng- ing due to the lack of contour annotations. Nevertheless, weakly supervised se- mantic segmentation methods are proficient in utilizing intra-class feature consis- tency to capture the boundary contours of the same semantic regions. In this paper. we design a Mutual Distillation Module (MDM) to leverage the complementary strengths of both instance position and semantic information and achieve accu- rate instance-level object perception. The MDM consists of Semantic to Instance (S2I) and Instance to Semantic (I2S). S2I is guided by the precise boundaries semantic regions to learn the association between annotated points and instance contours. 12S leverages discriminative relationships between instances to facilitate the differentiation of various objects within the semantic map. Extensive experi- ments substantiate the efficacy of MDM in fostering the synergy between instance and semantic information, consequently improving the quality of instance-level object representations. Our method achieves 55.7 mAP50 and 17.6 mAP on the PASCAL VOC and MS COCO datasets, significantly outperforming recent PSIS methods and several box instance segmentation competitors. Our code is available on https://gituubcccmmuua////////
INTRODUCTION
Instance segmentation is critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important. In recent years, instance segmentation has gained extensive interest due to its significance in various application scenarios and research domains (Dai al. 2016a 2015; 2016b, He al. 2017b] Kirillov et al.) 2017 Liu et al.) 2016 Novotny et al. 2018 and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate instances of the same category. The advanced instance segmentation methods use point annotations that only approximate object positions, making it difficult to capture detailed features and accurate boundaries. Hence, relying solely on point annotations has limitations that would not allow instance segmentation methods to provide precise object contours. On the other hand, semantic segmentation in computer vision involves image classification, object recognition, and semantic region boundary delineation, and its goal to divide the image into distinct regions and assign category labels. Although advanced semantic segmentation excels at precise semantic region boundaries, it often struggles with instance-level discrimination within the same category.
Semantic information struggles to define precise boundaries when multiple same-class objects are close or overlapping. For example, an image contains multiple bottles, semantic information meth- ods lumps them into one "bottle" region, unable to distinguish between different bottles. Therefore, considering the limitations both instance information and semantic information have, it can be con- cluded that instance information is vital for accurate discrimination among instances but is limited
Equaa contribution. Corresponding author.

---

Published as a conference paper at ICLR 2024
Input
Point Separatied with Features
(e) Local Reportse
Inppul
Missing Instance information
(6) Mur Boundary
+
+
Figure I: Two limitations of PSIS methods: Left: severe local responses caused by points separated
with image features. Right: ambiguous boundaries caused by missing instance information.
Input
Semantic Map
Instance Map
Semantic Map
Instance Map
Figure 2:
The illustration S2I and I2S Complementary Advantage. Semantic and instance infor- mation is mutually distilled, resulting in high-quality instance segmentation maps.
to only instance point annotations; semantic information excels scene understanding and object recognition but struggles with instance-level information. Both have their own merits. In summary, as shown in Fig. the two problems faced by Point-level Supervised Instance Segmentation (PSIS) methods are:
The local response caused by the separation of points from image features. Image features and point supervision are decoupled and heavily rely on off-the-shelf proposals for instance segmen- tation, resulting in localized outcomes.
The semantic segmentation estimation and instance differentiation are separated. Point supervi- sion underutilized in the interaction of semantic segmentation and instance information, result- ing in inaccurate instance estimation and challenges in distinguishing similar object boundaries.
In this paper, aim solve the above two problems through the Mutual Distillation Module (MDM) of the P2seg method, which leverages the advantages of instance and semantic segmenta- tion. We achieve this by employing the Semantie Instance (S21) and Instance to Semantic (I2S) modules within MDM for knowledge distillation. Specifically, our contributions are as follows:
We design an MDM module that mutually distills instance and semantic information to facilitate improved PSIS without relying on pre-trained proposals.
We develop the S2I method transfer semantic information to instances, guiding the network toward producing improved final instance segmentation labels. These enhanced labels lead to superior performance in the subsequent segmentor training.
We propose the I2S module that merges instance details with class map (semantic score predicted by HRNet (Sun et al.) 2019)) via affinity matrix. This integration embeds instance information into semantic context to facilitate mutual distillation and improve instance estimation accuracy.
2 RELATED WORK
Semantic Digmentation Most WSSS methods extract object location using CAM from CNNs with image-level labels. However, CAM lacks precise boundaries and insufficient for training semantic networks. ReCAM (Chen et al. 2022) refines CAM with SCE loss. CLIMS (Xie et al. 2022) uses a text-driven approach for better object activation. AFA (Ru
et al.
2022) adds attention affinity module in the transformer to refine CAM. MCTformer (Xu
et al.
2022) improves CAM using patch-level pairwise affinity and transformer. L2G (Jiag et al.
2022
learns comprehensive attention knowledge from multiple maps. (Kweon et al. (023) improves the quality of CAM by reducing the inferability between segments. WSSS focuses on lixel-level classification and cannot distinguish instances within the category.
2

---

Published as a conference paper at ICLR 2024
MDM stage I....,
Semantic Map
S2I
Seg map
Points
MDM
MDM
loss
Sigmoid
Sottmax Dim--llin
Semantic Map
Supervisiom
Segmentor
Affinity Map
Offset Map
Class Map
S21 Branch
125 Branch
The architecture of MDM. In S2I branch, instance segmentation map is generated from the results of semantic segmentation using the offset map. In I2S branch, semantic segmentation results are influenced by instance segmentation map using affinity matrix. Finally, instance segmentation map generated by the trained S2I branch of MDM used supervision for training the segmentor.
Figure 3:
Seakll-Supervised Instance Segmentation(WSII)) Yeakly-superviser instance segmentation
(WSIS) struggles with obtaining instance-level information from image-level labels. IRNet (Ahn ct al. 2019) learns class-agnostic instance maps and pixel pair similarities, deriving pseudo instance segmentation labels without predefined proposals extra supervision. In contrast, Label-PEnet (Ahn et al. 2019 Ge et al.] 2019) transforms image-level labels into pixel-wise labels through a cascaded pipeline where the modules in this pipeline share backbone network and use curricu- lum learning to enhance accuracy gradually. PDSL (Shen et al. 2021) unifies object detection and segmentation by integrating bottom-up
into top-down pipeline. The detection-ssemmentttion
separation allows both to perform better in their respective tasks.
Pointly-Supervised Instance Segmentation(PSII)) The aim of PSIS is to leverage point annota- tions for improved segmentation, and PSIS methods, such as WISE-Net (Laradji et al. 2020), use hant- To generate high quality mask proposals. WISE-Net trains L-Net with point anno- tations to locate objects and then uses E-Net to group pixels, generating high quality mask proposals. BESTIE (Kim et al., 2022) refines mask proposals using point supervision and instance-level cues for robust instance segmentation. It addresses semantic drift by using self-refinemet and instance- aware guidance and enhances segmentation accuracy by effectively integrating semantic and in- stance information, thus surpassing methods relying solely image-level labels. SAM (Kirillov) jet al.) 2023bb an interactive image segmentation framework that can segment specified objects in an image using coarse hints, such as points, bounding boxes, and textual descriptions. Point2Mask (Li et al. 2023) employed in panoramic segmentation tasks. It uses ground truth points as super- visory information generate pseudo-label masks, thereby reducing annotation costs.
Mutual Distillation. Knowledge distillation (Buciluà et al., 2006 Hinton et al. 2015) distills knowledge from a complex teacher model into simpler, compressed student model to achieve sat- isfactory performance. Mutual learning strategy (Zhang et al., 2018) involves collaboration among students, allowing them learn from each other. (Wu al. 20199 applies mutual learning to each block of a salient object detection backbone. (Zhang et al. 2019) suggests self-distillation within network. (Crowley et al. 2018) transforms teacher network into a student network. (Hou) et al. 2019) uses self-attention distillation for lane detection. (Yun et al., 2020) distills prediction distribution among same-labeled samples. (Wei al.) (2020) employs mutual learning for image classification with noisy labels, aiming to reduce network diversity during training. (Pang et al.) 2020) introduces an aggregate interaction module to handle feature fusion challenges. (Yuan et al, 2020 proposes teacher-free knowledge distillation with improved label smoothing regularization. Yang et al. 2021) introduces mutual learning by considering network width and resolution simul- taneously. Inspired by these methods, propose mutual distillation learning for PSIS. It achieves effective fusion and information promotion from instances and semantics, thereby improving the quality of segmentation results.
3

---

Published as a conference paper at ICLR 2024
3 METHOD
Overview. As shown in Fig. 3 our proposed MDM consists distillation from semantic to instance information and instance semantic information. In framework, HRNet serves as the backbone network for feature extraction, which following BESTIE, predicts offset maps, class maps, and instance affinity matrices separately. MDM is recurrent learning module. Initially, S2I branch obtains initial offset maps from the generated semantic segmentation map, serving as the initial instance segmentation la- bels. The offset maps predicted by HRNet con- nect with the semantic segmentation maps to produce new instance segmentation maps.
OFF
Lou S21
Then, the I2S branch uses the newly generated instance segmentation maps to compute the in- stance affinity matrix. A new semantic seg- mentation map is generated by multiplying the affinity matrix with the semantic scores (i.e., class map) predicted from HRNet. The instance segmentation map generated the final stage of MDM used as pseudo labels to supervise the learning of the segmentor.
Semantic
HRNct
Instance
Loss E2S
AFF
AFT
Figure 4: The abstract concept of semantic and
instance information interaction. Corresponding Fig. 3 the S21 branch is colored green, while the I2S branch is colored in blue.
3.1 SEMANTIC TO INSTANCE
As shown in Fig. regarding the information distilled from semantic instance, introduce the S2I branch that combines point annotations and weakly supervised semantic segmentation results to create initial instance segmentation labels. It also calculates instance-level offset maps measuring inter-pixel distances within the initial results. Initially, it identifies preliminary instance regions using connected component labeling based on the semantic segmentation. Then, these regions are matched with the set of point annotations E {ei, e..... ex where K denotes the number of point annotations. cases of conflicts (e.g., multiple annotations one region), a distance separation strategy is used. The distance is defined as: k" arg min |ex -pl
(1)
where p denotes the points in semantic segmentation map, eg denotes the k-th annotation point, k* represents the point p assigned to the k -th annotation point ex. After obtaining the initial
Points
Distince Separation Strategy
Points
Map
Semantic Map
Semantic
Map
Seg.
The architecture of the S21 branch. Left: The semantic segmentation map generates the offset map and class maps as supervised. Right: The process generating the new instance segmentation map.
Figure 5:
instance segmentation labels from the distance separation strategy. it generates the corresponding offset maps from the initial labels. Similar to BESTIE, we calculate the offset map for all instance points belonging to the annotation points by calculating the offset between the annotation and other points. The formula is as follows:
=
U
lle - m
ex
where m denotes the points within the instance region to which the k-th annotated point eg belongs. O denotes the offset maps.

---

Published as a conference paper at ICLR 2024
The offset maps generated by HRNet prediction is not only used for generating class class-agnostic in- stance segmentation maps based on points but also contributes to the calculation of the loss function with initial offset map to facilitate further network learning through gradient back- propaaationnnnn The class-agnostic instance segmentation map is multiplied with the semantic segmentation map gener- ated through I2S branch, leading new instance segmentation results. It is important to note that instance adaptive grouping is based on point annotations for mask grouping. Our network creates final instance segmentation labels based on combination of predicted semantic segmentation re- sults, instance similarity matrices, and point annotations. In case of discrepancies between semantic and point- based segmentation, point annotations take precedence. For points without clear instance predictions, a default pseudo-box of size 16x 16 is assigned as their segmentation prediction.
In the first stage, semantic map for S2I consists of pseudo-labels generated from the off-the-shelf segmentation map (e.g. SAM). Since the second stage, the semantic map input to MDM is generated from the previous stage. The instance segmentation map generated in the final stage will serve as pseudo-labels for the instance segmentor and perform the ultimate fine-tuning of the model.
3.2 INSTANCE TO SEMANTIC
As shown in the blue region in Fig. through the I2S branch, semantic information is distilled into instances. First, the branch obtains an affinity matrix from the instance segmentation map generated by S2I, representing the instance affinity between pairs of pixels. Specifically, in the instance similarity matrix, if two pixels belong to the same instance, their value is set to 1, otherwise, is set to O. Subsequently, this branch uses the affinity matrix A E Rhwxke predicted by HRNet as weights adjust the class map C E [ewx((+1) generated by HRNet. This adjustment enables the class map to incorporate instance information and generate the updated semantic segmentation map S = A C, where A denotes the / times Hadamard power of A, which smoothens the distribution to attain the optimal semantic segmentation map. Then affinity matrix generated from the instance segmentation map is used to compute the Lr2s (see Sec. S.c.3.3) with the predicted affinity from HRNet to facilitate gradient backpropagation and promote self- self-learning of the network. The semantic segmentation map obtained through I2S serves as the semantic information input for the next stage. Through the constraints of instances, I2S enriches the semantic segmentation results with instance information. 12S can be widely applied to various weakly supervised segmentation tasks to allow the segmentation prediction with iistancewwise information.
3.3 LOss
As shown in Fig. 3 our framework consists of two branches corresponding to the respective two loss functions, namely the S2I loss C521, the I2S loss L12s.
S2I Loss. S2I loss consists of the offset loss and the segmentation loss, i.e., Ls2s Lofs + Zeeg which is primarily used to constrain the network to capture instance information. Based the annotations, we generate the offset map of the pixel value of the corresponding instance object according to the point annotations, which is represented by vector (vr,)) y). Offset map collects sets of pixels for labeled instance regions from pseudo labels, and set is denoted as Preeddo The offset loss expressed as follows:
Coss
Σ
W(i.j) smooth -((,))
P (Ppecudo
(3)
(1.))
where predicted pseudos in the former stage are Ô, respectively. W means the weight of the pseudo loss. Offset loss represents the instance object's scale range that can well constrain the network regression. For semantic segmentation branch, the objective function is defined as:
Laeg
Σ CE(Peu,))C(i..
(4)
iPeegl
(t.))PP.eg
where é, C respectively represent the class map predicted by the network and the class map obtained the S2I stage while acquiring the offset map. Pê(a,)) represents the probabilities predicted by the network for each category in the class map, and Pang is the set of all pixels in ć.
I2S Loss. I2S loss uses the instance ownership relation between pixels to model the affinity matrix.
The expression of 12S loss is as follows:
5

---

Published as a conference paper at ICLR 2024
Input
SKT
Instance Map
Input
S2I
Instance Map
Accept
Accept
Figure 6: Comparison of distillation from S2I. The yellow area represents BESTIE method, and the green zone indicates the S2I branch.
Σ (2-0 - -10) N
Σ (a + (At)): (1
L1s5 N+
(5)
(LJ)
where a represents the sigmoid activation function. R+ and R. denote the set of affinity and non- affinity pixel-pair samples. (i, j), (ki, 1) denote the pixel pairs, respectively. N+ and N count the number of R+ and It, A and Á means the the affinity matrix generated from the instance map produced by the S2I process and predicted instance affinity matrix by the network HRNet. Eq. 5 reinforces the network to learn highly confident instance affinity relations from iterative training. On the other hand, since the instance affinity prediction is used by semantic segmentation branch, also benefits the learning of global instance features and further helps to discover the integral object regions. The overall loss is finally formulated as A12shr2s + As2 L521.
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
Datasets. Our experiments are conducted PASCAL VOC 2012 (Everingham al. 2010) and MS COCO 2017 (Lin et al. 2014) datasets. VOC dataset contains 20 instance classes, which is usually augmented with the SBD (Hariharan et al. 2011) dataset. It includes 10,582, 1,449, and 1, 1,464 images for training, validation, and testing, respectively. COCO dataset contains 80 classes and includes 18& images for training and 5k images for validation. The images we used for training annotated with point-level labels only.
Evaluation Metries. We assess the performance instance segmentation using two measures. We measure the performance using the standard protocol mean Average Precision(mAP), First, mea- sure the performance using standard protocol mean Average Precision(mAP) with intersection- over-union(loU) thresholds of 0.5, 0.7, and 0.75 for VOC. Second, the averaged AP over IoU thresh- olds from 0.5 to 0.95 is used on COCO dataset.
AdamW (Loshchlovv & Hutter 2017) optimizer to train our network. The initial learning rate is set as 5x 10-9 and the batch size is set as For the experiments on the VOC dataset, train the network for 50,000 iterations. To ensure the initial pseudo labels favorable, we warm-up the network for 2,000 iterations. For experiments on the COCO dataset, the total iterations number is 100,000 and the number of warm-up iterations is 5,000. The weights of semantic segmentation loss. offset map loss and instance affinity matrix loss are set as 1.0, 0,01. 1.0, respectively. The data augmentation strategies used here includes random resizing between 0.7 and 1.3, random cropping, random flipping, and photometric distortion. In the fine-tuning phase of the network's instance segmentor, we primarily employ the Mask R-CNN (He et al. (017a) to retrain the instance segmentation map obtained from the last stage of MDM. We use SGD optimizer with learning rate of 0.02 and the batch size is 16 with 12 epoches for both VOC and COCO dataset.
Training details. We
4.2 COMPARISON WITH STATE-OP-THE-AD METHODS.
Results on Pascal VOC 2012. We report the performance of the proposed method on the com- monly used VOC validation dataset( represents the method reproduced in this paper). As shown image-supervised box-supervised, and pointly-supervised
in Table I the methods grouped methods from top to bottom. Our method achieves the best performance among all competing meth- ods and even outperforms the most competitive BESTIE both mAPso 50 and mAP

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison of the state-of-the-art WSIS methods on VOC 2012 val-set. We denote the supervision sources as: F (full mask), B (box-level label), I (image-level label). P (point-level label), and C (object count). The off-the-shelf proposal techniques are denoted fol- lows: M (segment proposal (Pont-Tuset al., (016)). , R. (region proposal (Uijlings et al. 2013 and Sr. (salient instance segmentor (Fan et al. 2019
Method
Sup.
Backbone
Extra
mAP25 76.7
mAP6o
mAP70
mAP76
Mask R-CNN (He et 2017a
F
ResNet-50
67.9
44.9
End-to-Enn End weakly-supervised models.
I
ResNet-50
44.3
26.8
IAM
Zhu
I
ResNet-50
M
45.9
28.3
11.9
Jabel-PFnne (ie ct
2019
I
VGG-16
R
49.2
30.2
12.9
CL (Hwang ct al.
2021
I
ResNet-50
M.R
56,6
38.1
12.3
BBTP
(Hsu al.
2019
B
ResNet-101
23.1
54.1
17.1
BBTP W/C
B
ResNet-101
27.5
59.1
21.9
BoxInst
Tian ct al.
2021
B
ResNet-101
60.1
34.6
OCIS Chollkkal cl al.
2019
C
ResNet-50
M
48.5
30.2
14.4
Point2Mask
2023
P
ResNet-101
48.4
22.8
Sulti-Stuge
weakly-supervised models.
WISE Laradji et
2019
ResNet-50
MM
49.2
41.7
23.7
IRN
al.
I
ResNet-50
46.7
23.5
LIID
2020
I
ResNet-50
M.Sr
48.4
24.9
Arun ef
Arun
2020
I
ResNet-101
M
59.7
50.9
30.2
28.5
WISE-Net
Laradji
2020
P
ResNet-50
M
53.5
43.0
25.9
BESTIE!
Kim
2022
P
HR
62.8
52.8
31.2
SAM
Kirillov ct al.
2025a
P +
VIT-S/22.1M
59.4
39.9
19.0
Ours
P ResNet-101
64.1
53.9
37.7
32.0
Ours
P
HRNet-48
65.8
55.6
40.2
34,4
Table 2: Quantitative comparison of the state-of-the-art WSIS methods on MS COCO 2017 val-set. We denote the supervision sources as: FF (full mask), B(box-level label), I(image-level label), and P (point-level label). The off-the-sheeff proposal techniques are denoted as follows: M (segment proposal (Pont-Tuset et al. 2016)).
Method
Sup. F ResNet-50
Backbone
Extra
AP
AP50
AP75
Mask R-CNN (He et al. 2017a)
34.6
56.5
36.6
End- -nd weakly-supervised models.
BBTP
Hsu et al. 2019
B
ResNet-101
21.1
45.5
17.2
BoxInst
Tian al. 202
ResNet-101
31.6
54.0
31.9
Point2Mask (Li et al.
2023
P
ResNet-101
12.8
26.3
11.2
Multi-Stage weakly-supervised models.
IRN (Ahn et al. 2019
I
ResNet-50
6.1
11.7
5.5
WISE-Net
(Laradji et al..
2020
ResNet-50
MM
7.8
18.2
BESTIE
(Kim et al., 2022] P
HRNet-48
14.2
28.4
22.5
Ours
P
HRNct-48
17.6
33.6
28.1
Results on MS COCO 2017. The proposed P2Seg is compared with the state-of-the-art on more challenging COCO dataset( represents the method reproduced in this paper). In Table 2 our method achieves the best performance among competing methods and even outperforms the most competitive BESTIE both AP50 and AP75. We find that P2Seg significantly outperforms weakly supervised methods and on par with top-performing box-supervised methods.
Overall Analysis. In summary of all the above experimental results, P2Seg significantly enhances AP75, leading to overall segmentation performance improvement. The experiments illustrate that the mutual distillation method, leveraging instance and semantie information, can effectively esti- mate target scale and shape without requiring subsequent network fine-tuning. This optimization enhances handling of fine-grained segmentation details and resolves challenges with adjacent ob- jects of the same class, ultimately improving segmentation performance.
4.3 ABLATION STUDY AND ANALYSIS
In order to further investigate the effectiveness of the instance-basel and semantic- basse mutual distillation algorithm, this section will conduct corresponding ablation experiments the mutual

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study for Table 4: Ablation study for differ- our S2I and I2S, compared with ent Segmentor Backbones. BESTIE.
Table 5: Ablation experiment to analyze the impact of hard pixel ratio.
S-I
I->S
mAP50
Method BESTIE
Segmentor
mAP30
Hard pixel ratio
mAP50
52.8
0.1
52.0
BESTIE
52.8
Ours
55.6
0.2
50.8
S2I
53.2
BESTIE
51.9
0.4
51.5
S2I
12S
55.7
Ours
SOLOv2
54.1
0.8
50.9
Table 6: The comparison of BESTIE and our P2Seg for IoU with the ground truth.
Method
loU? 50
loU>70 70
loU> 90
overall loU
Semantic Results First
5782
4939
2440
58.49
Points First
9544
7417
2558
66.57
distillation method. Unless otherwise specified, the datasets used in this section are from the PAS- CAL VOC 2012 dataset.
Effect of S21. The motivation of S2I is to distill the semantic segmentation information into instance segmentation information. Based on the point annotation, S2I also shows its powerful ability to merge semantic segmentation results with point annotations. As shown in Fig. [ the yellow part illustrates the Semantic Knowledge Transfer method proposed in BESTIE, which has a similar role to the S2I proposed in this paper. However, BESTIE does not fully utilize existing point annotations, whose design only considers scenarios where point annotations and foreground regions have no conflicts, leading to the loss of adjacent objects. In contrast, S2I assigns unique instance region to represent each point's instance, ensuring better distinction and allocation of instances. As shown in Table E $21 obtains 1.4 mAP50 performance improvement PSIS tasks compared with the previous methods, showing the effectiveness of $2I statistically.
Effect of 12S. I2S module improves instance discrimination in the same class by using the instance affinity matrix. As shown in Table 3 I2S raises mAP50 from 53.2 to 55.7. The Hadamard power of the affinity matrix provides additional information.
Other Segmentors for Instance Segmentation. As shown in the Table we further evaluate instance segmentation networks trained with our pseudo labels, comparing them to SOLOv2 (Wang et al. 2020) and Mask R-CNN (He et al.) 2017a). which are effective networks for single-stage and two-stage methods, respectively. The table displays results of Mask R-CNN and SOLOv2 trained with our pseudo labels, showing significant improvement over the previous state-of sate-off-hhe-rr BESTIE. Our method outperforms BESTIE by 2.8 mAP50 on Mask R-CNN and 2.2 mAP on SOLOv2, highlighting the superiority of mutual distillation over one-way distillation.
Hard pixel ratio. As shown in Table 5 the term "hard pixel ratio" refers to the proportion of challenging samples used in loss computation. A lower proportion of hard pixels during training yields higher mAPs0. which is the validated result without a segmentor. However, reducing this easier examples and limit the model's ability to handle diverse
proportion can lead to overfitting scenarios, resulting in decreased generalization performance. We set the hard pixel ratio to 0.2.
Prediction Mask Quality Comparison. Considering that the post-proceet processiin in this approach in- volves fine-tuning the Mask RCNN network, this study further evaluates the quality of the predicted masks obtained before inputting them into the Mask RCNN network. o PASCAL VOC 2012 dataset, showing the number of predicted masks with loU thresholds of 0.5, 0.7, and 0.9 compared to ground truth masks in the first three columns. The "average IoU" column presents the mean IoU between predicted instance masks and ground truth masks.
data from the
Backbone, We make the ablation experiments about our method on the backbone layer of the network. As shown in Table [ our method has consistent effects in different feature extracting methods from the backbone network.
4.4 VISUALIZATION
As shown in Fig. the results demonstrate image segments using our method and the baseline. P2Seg effectively distinguishes boundaries between neighboring objects in the same class to enhance
8

---

Published as a conference paper at ICLR 2024
Figure 7: Visualization result on COCO. First row: Original image. Second row: Segmentation results of BESTIE before fine-tuning. Third row: Segmentation results of the point-supervised instance segmentation network constructed in this paper before fine-tuning. Fourth row: GT. w/o Mutual Distillation
Image
initial
5000 iter
10000% iter
15000 iter
20000 iter
500007 iter
w/ Mutual Distillation
Label
initial
5000 iter
10000 iter
1.50009 iter
200000*
5000000 iter
Figure 8: Comparison of class-agnostic instance segmentation learning. Left: Original image and class-agnostic instance segmentation annotations. Right: Learning progression from initialization 50,000 iterations. The top without mutual distillation, and the bottom with
completeness and boundary clarity. The study validates the impact of mutual distillation instance features through visualizations of agnsstii instance segmentation maps. As shown Fig. [ compare the segmentation results with and without mutual distillation reveals that the network gains more instance information over time with MDM, allowing it to segment more objects. MDM method significantly enhances the point-supervieed instance segmentation network.
CONCLUSION
We propose MDM, a mutual distillation network using point-level data. It effectively combines point annotations with features and facilitates mutual distillation of information between instance- level and semantic-level information, and distinguishes object boundaries within the same class. MDM employs S2I and I2S processes, enhancing simple class and localization attentions for fully supervised instance segmentation training. Trained on Pascal VOC 2012, MDM shows outstand- ing performance in both instance and semantic segmentation tasks, showing the power of mutual distillation.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported in part by the Youth Innovation Promotion Association CAS, the National Natural Science Foundation of China (NSFC) under Grant No. 61836012 and 61771447, the Strate- gic Priority Research Program of the Chinese Academy of Sciences under Grant No.XDD27000000.
REFERENCES
Jiwoon Ahn, Sunghyun Cho, Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2204-2213, 2019.
Aditya Arun, CV Jawahar, and M Pawan Kumar. Weakly supervised instance segmentation by learning anno- tation consistent instances. In European Conference Computer Vision, pp. 254-270 Springer, 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Siculescu-Mizii Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541 2006.
Zhaozheng Chen, Tan Wang, Xiongwei Wu, Xian-Sheng Hua, Hanwang Zhang, and Qianru Sun. Class re- activation maps for weakly-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 969-978, 2022.
Hisham Cholakkal, Guolei Sun, Fahad Shahbaz Khan, and Ling Shao. Object counting and instance segmen- tation with image-level supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp- 12397-12405, 2019.
Elliot Crowley, Gavin Gray, and Amos Storkey. Moonshine: Distilling with cheap convolutions. Advances in Neural Information Processing Systems, 31, 2018.
J. Dai, K. He, and J. Sun. Convolutional feature masking for joint object and stuff segmentation. In Computer Vision and Pattern Recognition, 2015.
J. Dai, K. He. Li S. Ren, and J. Sun. Instance-sensitive fully convolutional networks. arXiv e-prints, 2016a.
Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3150-3158, 2016b.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
Ruochen Fan, Qibin Hou, Ming-Ming Cheng, Gang Yu, Ralph Martin, and Shi-Min Hu. Associating inter- image salient instances for weakly supervised semantic segmentation. In Proceedings of the European conference computer vision (ECCV), pp. pp. 367-383, 2018.
Ruochen Fan, Ming-Ming Cheng, Qibin Hou, Tai-Jiang Mu, Jingdong Wang, and Shi-Min Hu. S4net: Single stage salient-instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 6103-6112, 2019.
Weifeng Ge, Sheng Guo, Weilin Huang, and Matthe Scott. Label-penet: Sequential label propagation and enhancement networks for weakly supervised instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3345 3354, 2019.
Bharath Hariharan. Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji. and Jitendra Malik. Semantic contours from inverse detectors. In 2011 international conference computer vision, pp. 991-998. IEEE, 2011.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 201
Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In JCCV, 2017b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255: 2015.
Yuenan Hou, Zheng Ma. Chunxiao Liu, and Chen Change Loy. Learning to steer by mimicking features from heterogeneous auxiliary networks. In Proceedings the AAAI Conference Artificial Intelligence, volume 33, pp. 8433-8440, 2019.
10

---

Published as a conference paper at ICLR 2024
Cheng-Chun Hsu, Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, and Yung-Yu Yu Chuang. Weakly supervised instance segmentation using the bounding box tightness prior. Advances Neural Information Processing Systems, 32, 2019.
Jaedong Hwang, Scobyun Kim, Jeany Son, and Bohyung Han. Weakly supervised instance segmentation by deep community learning. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 1020-1029, 2021.
Peng- Jiang, Yuqi Yang. Qibin Hou, and Yunchao Wei. 1.2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16886-16896, 2022.
Beomyoung Kim, Young Joon Yoo, Chac-Eus Rhee, and Junmo Kim. Beyond semantic to instance segmenta- tion: Weakly-supervisce instance segmentation via semantic knowledge transfer and self-refinement, 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 4268 4277, 2022.
A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and C. Rother. Instancecut: From edges to instances with multicut. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
Alexander Kirillav, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao,
Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. arXiv:230.002643 2023a.
Segment anything. arXiv preprint
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.02 02643, 2023b.
Hyeokjun Kweon, Sung Hoon Yoon, and Kuk-Jin Yoon. Weakly supervised semantic segmentation via adver- sarial learning of classifier and reconstructor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11529-11339, 2023.
Issam H Laradji, David Vazquez, and Mark Schmidt. Where the masks: Instance segmentation with image- level supervision. arXiv preprint 1rXiv:1907,014533 2019.
Issam H Laradji, Negar Rostamzadeh, Pedro Pinheiro, David Vazquez, and Mark Schmidt. Proposal-based instance segmentation with point supervision. In 2020 IEEE International Conference on Image Processing (ICIP). pp. 2126-2011 2130. IEEE, 2020.
Wentong Li, Yuqian Yuan, Song Wang, Jianke Zhu, Jianshu Li, Jian Liu, and Lei Zhang. Point2mask: Point- supervised panoptic segmentation via optimal transport. arXiv preprint arXiv:2308.0177 2023.
Yi Li, Zhanghui Kuang, Liyang Liu, Yimin Chen, and Wayne Zhang. Pseudo-mask matters in weakly- supervised semantic segmentation. Proceedings of the IEFF/CVF international conference on computer vision, pp. 6964-6973, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer. 2014.
S. Liu, X. Qi. J. Shi, H. Zhang, and J. Jia. Multi-scale patch aggregation (mpa) for simultancous detection and segmentation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yun Liu, Yu-Huan Wu, Pei-Song Wen, Yu-Jun Shi, Yu Qiu, and Ming-Ming Cheng. Leveraging instance-, image-and level information for weakly supervised instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Tlya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/171 1.05101, 2017.
David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Semi-convolutional operators for instance segmentation. In Proceedings of the Luropean Conference Computer Vision (ECCV), pp. 86-102, 2018.
Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Multi-scale interactive network for salient object detection. In P Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pp. 2020.
Jordi Pont- Tuset, Pablo Arbelacz, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale com- binatorial grouping for image segmentation and object proposal generation. IEEE transactions pattern analysis and machine intelligence, 1981)122-114,, 2016.

---

Published as a conference paper at ICLR 2024
Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du Learning affinity from attention: End-to-end weakly- supervised semantic segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16846-1660 16855, 2022.
Yunhang Shen, Liujuan Cao, Zhiwei Chen, Baochang Zhang. Chi Su, Yongjian Wu, Feiyue Huang, and Ron- grong Parallel sareriionnnnd-segmmeee IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8178-8188, 2021.
learning for weakly supervised instance segmentation. 202/
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5693-5703, 2019.
Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: High-performance instance segmentation with box annotations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 5443-5452, 2021.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104:154-171, 2013.
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. Advances in Neural information processing systems, 172171111777,, 2020.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13726-13735, 2020.
Runmin Wu, Mengyang Feng, Wenlong Guan, Dong Wang, Huchuan La, and Errui Ding. A mutual learning method for salient object detection with intertwined multi-supervision. Proceediggs of the IFFF/CVF conference on computer vision and pattem recognition, pp. 81 8150-8159, 2019.
Jinheng Xie, Xianxu Hou, Kai Ye, and Linlin Shen. Cross language image matching for weakly supervised semantic segmentation. arXiv preprint arXiv:2203.026m: 2022.
Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4310-4319, 2022.
Taojiannan Yang, Sijie Zhu, Matias Mendieta, Pu Wang, Ravikumar Balakrishnan, Minwoo Lee, Tao Han, Muharak Shah, and Chen Chen. Mutualnet: Adaptive convnet via mutual learning from different model configurations. IEFF. Transactions on Pattern Analysis and Machine Intelligence, 45(1):811-827, 2021. 2021.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self- knowledge distillation. Proceedings of the IEEE/CVF conference on computer vision and puttern recog- nition, pp. 13876-13885, 2020.
I Linfeng Zhang, Jicbo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 3713-3722, 2019.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320-432545 -328, 2018.
Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Weakly supervised instance segmentation us- ing class peak response. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3791-3800, 2018
Yi Zhu, Yanzhao Zhou, Huijuan Xu, Qixiang Ye, David Doermann. and Jianbin Jiao. Learning instance acti- vation maps for weakly supervised instance segmentation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 3116-3125, 2019.
12


Reviews:
[{"review_id": "agjHEojgbp", "reviewer": "Reviewer_enoR", "summary": "This paper addresses Point-level Supervised Instance Segmentation (PSIS). The authors argue that the existing PSIS methods usually suffer from the lack of contour annotations, and thereby precisely predicting boundaries is still challenging. As a remedy, this paper introduces the Mutual Distillation Module (MDM), leveraging the complementary benefits of semantic information and instance position. In MDM, Semantic to Instance (S2I) exploits the precise boundary information of semantic maps to enhance the instance contours. Meanwhile, Istance to Semantic (I2S) uses discriminative instances to differentiate the number of objects in the semantic map. Extensive experiments and comparisons are conducted on the PASCAL VOC and MS COCO datasets.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The idea is sounding.\n2. The ablations studies are extensively conducted.", "weaknesses": "[Major]\n1. Some important and recent WSSS methods are missing. Please refer to the following CVPR 2023 papers.\n* Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor\n* Out-of-Candidate Rectification for Weakly Supervised Semantic Segmentation\n* Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation\n* Token Contrast for Weakly-Supervised Semantic Segmentation\n\n2. The performance table is incomplete. First, please show the result of mAP25. Also, there have been several works using the transformer backbone, such as Point2Mask or AttentionShift. Please compare with them. In addition, where is BESTIE (with Res101)? Finally, please include the results of the COCO test set.\n\n3. The segment-anything model (SAM) can be an excellent option for solving PSIS. Please compare the proposed method with the SAM, from the performance perspective.\n\n[Minor]\nThe overall  writing should be enhanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E1N32lxtg2", "reviewer": "Reviewer_fR5S", "summary": "The paper proposed a novel approach “mutual distillation” for Point Supervised Instance Segmentation (PSIS). The method utilizes point-supervised semantic segmentation results as an initialization for guiding instance segmentation, and then use an affinity matrix that represents instance segmentation details to optimize the semantic information This process achieves mutual distillation between instance and semantic information to improve the final result of instance segmentation. They validated the effectiveness of the proposed method on the VOC and COCO datasets.", "questions": "Please see the detailed information in the weakness part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe concept of \"MUTUAL DISTILLATION\" in point instance segmentation is both novel and interesting. In my experimental observations, BESTIE doesn't handle adjacent objects well, possibly because it relies solely on semantic segmentation results without fully utilizing point-represented instance information. The motivation behind this paper aligns with my observations, and therefore, the proposed concept of mutual information exchange between semantic and instance information seems sound to me.\n2.\tThe paper's experimental section is comprehensive, using two datasets, different backbones, segmentation architectures, and conducting essential ablation studies.\n3. It appears that the visualization is effectively presented.", "weaknesses": "1.\tObservations from Figure 7 and Figure 8 suggest that the proposed mutual distillation method appears to perform well in segmenting adjacent objects and addressing missing object issues. It would greatly enhance the quality of this paper if the authors could provide a quantitative analysis of these cases.\n2.\tMinor issue\nThe writing of introduction should be improved somewhat.\n-  \"Instance segmentation is a critical task in computer vision and is equally important in semantic segmentation estimation and instance discrimination.\"\nto \n\"Instance segmentation is a critical task in computer vision, where semantic segmentation estimation and instance discrimination are equally important .\"\n- “and it aims not only to locate objects accurately but also estimate their boundaries to differentiate”\nto\n“and it aims not only to locate objects accurately but also to estimate their boundaries to differentiate”\nI understand the description in 2nd paragraph of introduction. But the 术语 “semantic segmentation” and “instance segmentation” may not suitbale, it should be “semantic information“ and “instance information”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFhxhyCUnm", "reviewer": "Reviewer_ZJkY", "summary": "This paper proposes a Mutual Distillation Module (MDM) to leverage the complementary strengths of both instance position and semantic information and achieve accurate instance-level object perception. It consists of Semantic to Instance (S2I) and Instance to Semantic (I2S) sepcifically.", "questions": "Please refer to the weakness part. I will adjust my rating based on the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This method is novel and new. Training in a multi-stage way, combining the advantages of both semantic segmentation and instance segmentation looks reasonable. The experimental improvement also looks good.", "weaknesses": "1. Training in a multi-stage way, I feel concerned about the efficiency of the method. The author should provide comparisons about efficiency, like GFLOPS, FPS, inference time and so on, with existing methods, to demonstrate that the improvement does not come from extra computational budget. In addition, will multi-stage training derive into more training iteration numbers? If so, the author should also provide fair comparison with the same iteration numbers.\n2. The ablation study is conducted on VOC, where S2I looks like a 0.4% improvement. The effect of S2I is thus quite limited. I am also curious about the ablation study on the COCO dataset. Will the improvement becomes less? If se, I feel that S2I is unnessary and extra.\n3. What's about the performance of the method on the COCO dataset with ResNet?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWARqYM7QC", "reviewer": "Reviewer_3EDk", "summary": "This paper focuses on the problems of local response and seprated semantic and instance learning on the pointly-supervised instance segmentation task. To solve the problems, the authors propose the Mutual Distillation Module, which achieves the conversion and cooperation between semantic and instance segmentation by predicting class, offset, and affinity, thus leveraging the complementary strengths of instance and semantic. Various experiments demonstrate the effective of the proposed method.", "questions": "**  Could the author explain the details described in Weaknesses 1?\n\n**  What impact do the hyperparameter β and the number of MDM stages have on performance?\n\n**  Does the use of SAM destroy fair comparisons, and would other pseudo-label generators be useful?\n\n**   Will the proposed S2I method be sensitive to the position of the points?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "** The analysis of the complementarity of instance segmentation and semantic segmentation is reasonable for the pointly-supervised instance segmentation task, and the designed method corresponds to the analysis.\n\n** The core S2I method is unique and more accurate than BESTIE.\n\n** Experiments demonstrate that both the Semantic to Instance (S2I) and Istance to Semantic (I2S) module is effective and the overall method outperforms previous works.", "weaknesses": "**  Writing and presentation need improvement. For example, there are some confusing descriptions that are not explained (e.g., Dim-align in Fig. 3, instance adaptive grouping in Sec. 3.1, the green arrow from S2I loss to the mask in Fig. 3).\n\n**  Some key technical details are missing. In Fig. 3, there are multiple MDM stages, but the required number of stages are not specified or experimented. the additional training cost should be discussed.\n\n** The ablation about β in Sec. 3.2 are missing, so the effect on “smoothening the distribution to attain the optimal semantic segmentation map” is not clear.\n\n** According to Sec. 3.1, the pseudo labels used in the first stage are obtained by SAM, but SAM is contrary to the setting of weak supervision. Moreover, in Tab. 1 and 2 the authors mark the proposed method as “no extra data”. But in my opinion, the used off-the-shelf segmentation map should be considered as extra data.\n\n**  Will the proposed S2I method be sensitive to the position of the points? Could the author provide some qualitative or quantitative analysis to further illustrate the robustness of the method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zi5FviC7T6", "reviewer": "Reviewer_QcUt", "summary": "The paper presents an approach to Point Level Supervised Instance Segmentation based on mutual learning and knowledge distillation. The proposed algorithm, P2seg, introduces mutual distillation to recover instance segmentation based on points supervision. Comparison with the state-of-the-art methods is done using Pascal Voc-2012 and COCO-2017.", "questions": "Please explain:\n\n    how the semantic segmentation is obtained.\n    Why BESTIE is retrained, for both PascalVOC-2012 and COCO-2017.\n    Why so many iterations are used, and what is included.\n    Provide details about the number of points used and the relations points-targets.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Originality. Point-supervised instance segmentation is not new in the weakly supervised literature, most approaches being more oriented to panoptic segmentation, ensuring a parsing of scenes better than instance segmentation. The novelty here is to use mutual learning between semantic segmentation and instance segmentation. The mutual information module is formed by a network S2I that transfers the semantic information to instances and a module I2S that uses the instance affinity matrices, class maps and offsets map predicted by an off-the-shelf segmentation network, namely HRNet (see e.g. \\cite{ YuanCW19} and (https://github.com/HRNet/HRNet-Semantic-Segmentation) to refine results cyclically. \n\nQuality and clarity. The effort of explaining the architecture with a number of images and sketches is appreciable. \n\nSignificance. The proposed mutual information-based method could, in principle, be lifted to large-scale datasets.", "weaknesses": "Novelty:\\\nThe paper is quite similar to BESTIE [Kim et Al., CVPR 2022], with the mutual distillation mimicking the semantic knowledge transfer defined in BESTIE. Many passages are pretty similar, though BESTIE provides algorithms and thorough explanations of the complex architecture, which are not given here.\n\nOrganization:\\\n-No information is given on the instance affinity matrix and class map, as computed by HRNet (as stated in the last line of pg. 3, Overview of Method). HRNet is a semantic segmentation net \\cite{YuanCW19}, which is never cited, and its practical role is not defined. \n\n-No ablation is provided to understand the effect of each component. For example, it needs to be clarified if the semantic segmentation map is obtained by full supervision or by weak supervision. The mIoU of the obtained semantic segmentation is not provided.\\\n-The number of points supplied at input, how they are collected, and the relation between them and the targets should be discussed.\\\nThe number of iterations used seems to be quite high.\n\nIn particular: \\\n A table showing scores, according to the number of points taken as input, was expected. Table 5, named “hard pixel ratio”, does not seem to be informative. \\\nA table showing the number of parameters for HRNet, MDM, and Mask-RCNN, used to retrain the MDM, was also expected, given the number of modules used.  \\\nA table showing the impact of semantic segmentation was expected.\\\n\nAmbiguities:\n- The full loss L = \\lambda_{I2S} L_{I2S} + \\lambda_{S2I} L_{S2I} is such that L_{S2I} = L_{offset} + L_{segmentation}, though no weight for balancing L_{S2I} is added. \nHowever, on page 6, paragraph Training details, it is written that the weight of the segmentation loss is 1.0 and the weight of the offset loss is 0.01, which do not sum to 1 for L_{S2I}.\n- As stated in Training details, Mask R-CNN is used for retraining MDM. Table 4 result is, in fact, the same as the result given in Table 1.\\\nHowever, in Table 1, BESTIE (points) is given on PascalVOC-2012 a mAP_{50} score of 52.8, while on BESTIE paper is reported 56.0, higher than the 55.6 mAP_{50} of the paper under revision. Likewise, there is no value for BESTIE mAP_{70}, which instead is 36.5. On COCO-2017 BESTIE obtains 34.0, while the results displayed here, on Table 2, are 33.6, therefore less than BESTIE. \n- In the two paragraphs “Results on Pascal VOC 2012” and “Results on MS COCO 2017”, it is written that the BESTIE algorithm is retrained to justify the lower scores reported, but it is not discussed why; while being the method almost similar an explanation is required. Therefore, apparently, the presented work does not improve on the SOTA. \\\nFurthermore, the works of \\cite{Kim_2023_CVPR} and \\cite{liao2023attentionshift} are not even reported, providing better results than those displayed in the paper under revision. \n\n- Possibly incorrectly reported values:\nThe difference between panoptic segmentation and instance segmentation is that instance segmentation does not consider ‘stuff’, and the metric mAP on ‘things’ allows confidence on overlapping objects. The difference implies (see \\cite{Kirillov_2019_CVPR}) that the metric PQ^{th}, that is panoptic quality on ‘things’, is like AP when from AP the non-overlapping predictions are subtracted, which means that AP benefits from predicting multiple overlapping hypotheses.\\\nThis paper, under revision, reports on Table 1 and 2 comparisons with Point2Mask \\cite{ li2023point2mask}) quite inexactly. \\\nIn fact, Point2Mask obtains on VOC 2012 results with PQ^{th}, namely panoptic quality on ‘things’, equal to 59.4 (with swin-transformer), and 53.0 with Resnet-101 while on Table 1 it is reported as mAP_{50} = 48.4 and mAP_{75} = 22.8. Similarly, on COCO-2017, no justification for these reported results is given.\n\n\nOther comments:\n1. Hadarmad should be Hadamard, which is repeated twice on pages 5 and 8.\n2. Pg. 5: The concept of “instance ownership relation” is not introduced.\n3. Pg. 5, eq. (5), it should be noted that A is assumed to be generated by the network HRNet, which is not referred to and it is a segmentation network. \n4. Pg 6: The description of COCO-2017 is wrong. It is written that COCO (i.e. COCO 2017) includes 110k images, but COCO 2017 includes 118k images. The test set is not reported. This is made of 40670 images, does this imply that no tests were made? Actually, there are no results on the test set.\n5. Pg. 6 declares, “We assess the performance of object detection using two measures. We measure the performance using the standard protocol mean Average Precision(mAP). “Object detection”? The performance should be about instance segmentation unless it starts with HRNet object detection. \n \n\nReferences\\\n@article{YuanCW19,\\\n  title={Object-Contextual Representations for Semantic Segmentation},\\\n  author={Yuhui Yuan and Xilin Chen and Jingdong Wang},\\\n  booktitle={ECCV},\\\n  year={2020}\\\n}\\\n@inproceedings{li2023point2mask,\\\n  title={Point2mask: Point-supervised panoptic segmentation via optimal transport},\\\n  author={Li, Wentong and Yuan, Yuqian and Wang, Song and Zhu, Jianke and Li, Jianshu and Liu, Jian and Zhang, Lei},\\\n  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\\\n  pages={572--581},\\\n  year={2023}\\\n \n@InProceedings{Kirillov_2019_CVPR,\\\n author = {Kirillov, Alexander and He, Kaiming and Girshick, Ross and Rother, Carsten and Dollar, Piotr},\\\n title = {Panoptic Segmentation},\\\n booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\\\n month = {June},\\\n year = {2019}\\\n }\\\n@inproceedings{fan2022pointly,\\\n  title={Pointly-supervised panoptic segmentation},\\\n  author={Fan, Junsong and Zhang, Zhaoxiang and Tan, Tieniu},\\\n  booktitle={European Conference on Computer Vision},\\\n  pages={319--336},\\\n  year={2022},\\\n  organization={Springer}\\\n}\\\n@inproceedings{liao2023attentionshift,\\\n  title={AttentionShift: Iteratively Estimated Part-Based Attention Map for Pointly Supervised Instance Segmentation},\\\n  author={Liao, Mingxiang and Guo, Zonghao and Wang, Yuze and Yuan, Peng and Feng, Bailan and Wan, Fang},\\\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\\\n  pages={19519--19528},\\\nyear={2023}\\\n}\\\n \n@InProceedings{Kim_2023_CVPR,\\\n author = {Kim, Beomyoung and Jeong, Joonhyun and Han, Dongyoon and Hwang, Sung Ju}, \\\ntitle = {The Devil Is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation}, \\\nbooktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, \\\nmonth = {June}, year = {2023}, pages = {11360-11370} }\\", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)