Single-Stage Open-world Instance Segmentation with Cross-task Consistency Regularization
Authors: XIZHE XUE
Dongdong Yu
Lingqiao Liu
Yu Liu
Satoshi Tsutsui
Ying Li
Zehuan Yuan
Ping Song
Mike Zheng Shou
Venue: Submitted to ICLR 2023 (2023)
Keywords: Open world
instance segmentation
Cross-task Consistency Regularization

Abstract: Open-World Instance Segmentation (OWIS) is an emerging research topic that aims to segment class-agnostic object instances from images. The mainstream approaches use a two-stage segmentation framework, which first locates the candidate object bounding boxes and then performs instance segmentation. In this work, we instead promote a single-stage framework for OWIS. We argue that the end-to-end training process in the single-stage framework can be more convenient for directly regularizing the localization of class-agnostic object pixels. Based on the single-stage instance segmentation framework, we propose a regularization model to predict foreground pixels and use its relation to instance segmentation to construct a cross-task consistency loss. We show that such a consistency loss could alleviate the problem of incomplete instance annotation -- a common problem in the existing OWIS datasets. We also show that the proposed loss lends itself to an effective solution to semi-supervised OWIS that could be considered an extreme case that all object annotations are absent for some images. Our extensive experiments demonstrate that the proposed method achieves impressive results in both fully-supervised and semi-supervised settings. Compared to SOTA methods, the proposed method significantly improves the $AP_{100}$ score by 4.75\% in UVO$\rightarrow$UVO setting and 4.05\% in COCO$\rightarrow$UVO setting. In the case of semi-supervised learning, our model learned with only 30\% labeled data, even outperforms its fully-supervised counterpart with 50\% labeled data. The code will be released soon.

Under review as a conference paper at ICLR 2023
TRANSFORMER-BASED OPEN-WORLD INSTANCE SeG- MENTATION WITH CROSS- TASK CONSISTENCY REGU-
LARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Open-Worke Instance Segmentation (OWIS) is emerging research topic that aims segment class-agnostic object instances from images. The mainstream approaches use a two-stage segmentation framework, which first locates the candi- date object bounding boxes and then performs instance segmentation. this work, we instead promote single-stage transformer-based framework for OWIS. We argue that the end-to-end training process in the single-stage framework can be more convenient for directly regularizing the localization of class-agnosti object pixels. Based on the transformer- -baed instance segmentation framework, we propose regularization model to predict foreground pixels and use its relation instance segmentation to construct a cross-task consistency loss. We show that such consistency loss could alleviate the problem of incomplete instance annotation a common problem in the existing OWIS datasets. We also show that the proposed loss lends itself to an effective solution to semi-super OWIS that could be considered an extreme case that all object annotations are absent for some images. Our extensive experiments demonstrate that the proposed method achieves impres- sive results in both fully-supervised and semi-supervised settings. Compared to SOTA methods, the proposed method significantly improves the APo00 score by 4.75% in UVO dataset +VVO dataset setting and 4.05% in COCO dataset -UVO dataset setting. In the case of semi-supervised learning, our model learned with only 30% labeled data, labeled data. The code will be released soon.
outperforms its sully-sup counterpart with 50%
INTRODUCTION
Traditional instance segmentation Lin al. (2014); Cordts al. (2016) methods often assume that objects in images can be categorized into a finite set of predefined classes (i.e., closed-world). Such an assumption, however, can be easily violated in many real-world applications, where models will encounter many new object classes that never appeared in the training data. Therefore, researchers recently attempted to tackle the problem of Open-World World Instance Segmentation (OWIS) Wang et al. (02TT. which targets class-agnostic segmentation of all objects in the image.
Prior to this paper, most existing methods for OWIS are of two-stage Wang et al. 2022) Saito et al. (2021), which detect bounding boxes of objects and then segment them. Despite their promising performances, such a paradigm cannot handle and recover if object bounding boxes are not detected. In contrast, a transformer- based approach called Mask2Former (Cheng et al. (2022) has recently been introduced, yet only for closed-word world instance segmentation. Based the Mask2Former, we propose a Transformer- Open-world Instance Segmentation method named TOIS.
Note that our work is not just straightforward adaptation of Mask2Former from close-world to open-world. This is because unlike closed-world segmentation, where the object categories can be clearly defined before annotation, the open-world scenario makes it challenging for annotators to label all instances completely or ensure annotation consistency across different images because they cannot have well-defined finite set of object categories. As shown in Figure aa), annotators miss some instances. It still remains challenging that how handle such incomplete annotations (i.e. some instances missed).

---

Under review as a conference paper at ICLR 2023
Mask
Foreground map
[X]
wrrmppete imneration en COEO IVV
(b) Metiatii of per rovel repiersation modile
(c) Vincaliantio cred 5015
APHaSS
AP10%%
8.4
MartRON
LOET
LGET
24
8.2
Markiferree
TON
54
$3.48
8.0
25:22
20
AR10915
7.8
24
16
13
7.6
+4.33
13:41
5525
TOIS Mask2For
10
18
40
60
50
(4 UVV--UUVV reutie
Chrraaapp Saaraaappp--Seplllle nordth
( cOcD-5tVV
(0,0000 roulis with voring arnotated cissas COCO
Figure I: (a). Instances missing annotations in COCO and UVO datasets. The regions in red boxes are mistakenly anmotated as background. (b). Motivation of our novel reg module (The consistency relationship between instance mask and foreground map). (c). Visualization results of our TOIS on UVO dataset. Here, the proposed TOIS is trained on COCO dataset and tested on UVO dataset. Our methods correctly segments many objects that are labeled in COCO. (d f). The AP100 of our TOIS vs. VOTAOTA methods on COCO-+UVO, Cityscapes->ae Paasscapes-Mapilllee -appillayy COCO-+VO, (g). The AR100% of our TOIS vs. baseline Mask2Former on COCO. From right left, with the total number of classes decreases (i.e. more instance annotations missed), the gain of our TOIS over baseline becomes larger, thanks to the capability of our model to handle incomplete annotations.
Recent work LDET Saito et al. (2021) addresses this problem by generating synthetic data with plain background, based on decoupled training strategy that can only be used in the two-stage method while our method is of single-stage. Another work called GGN Wang et al. (2022) handles such incomplete instance- annotation issue by training a pairwise affinity predictor for generating pseudo labels. But training such additional predictor is complicated and time-consuming
In contrast, our proposed TOIS method is end-to-end and simpler. We address this incomplete annotation issue via novel regularization module, which is simple yet effective. Specifically, it is convenient to concurrentyy predict not only (1) instance masks but also a (2) foreground map. Ideally, shown in Figure the foreground region should be consistent with the union of all instance masks. To penalize their inconsistency, we devise cross-task consistency loss, which can down-weight the adverse effects caused by incomplete annotation. This because when an instance is missed in annotation, long as it is captured by both our predictions of instance masks and foreground map, the consistency loss would be low and hence encourage such prediction. when annotations miss
Experiments in Figure T[g) show that such consistency loss is effective many instances. And as in Figure Tc) novel objects which are unannotated in training set have been segmented successfully by method.
So far, like most existing methods, we focus on the fully-supervised OWIS. In this paper, we further extend OWIS the semi-supervised setting, where some training images do not have any annotations at all. This is of great interest because annotating segmentation map is very costly. Notably. our proposed regularization module can also benefit semi-supervised OWIS consider unlabeled image an extreme case of incomplete annotation where all of the instance annotations are missed. Specifically, we perform semi-supervised OWIS by first warming up the network on the labeled set and then continuing training it with the cross-task consistency loss on the mixture of labeled and unlabeled images.
Contributions, In nutshell, main contributions could be summarized as:
2

---

Under review as a conference paper at ICLR 2023
1. Building upon a receenty-proposeee close-world segmentation method of Mask2Former, we propose a Transformer- based Open-world Instance Segmentation (TOIS) method.
2. We propose a novel cross-task consistency loss that mitigates the issue of incomplete mask annotations, which is critical issue for open-world segmentation in particular.
3. We further extend the proposed method into emmissupee OWIS model, which effectively makes use of the unlabeled images to help the OWIS model training.
4. Our extensive experiments demonstrate that the proposed method reaches the leading OWIS performance in the fully-supervised learning. (Figure [[d-f)) and that our semi-supervised extension can achieve remarkable performance with a much smaller amount of labeled data.
RELATED WORK
Closed-world instance segmentation (CWIS) He et al. (2017); Chen et al. (2020); Dai et al.
(2016); Bolya et al. (019); Wang et al. (2020a) requires the approaches to assign a class label and instance ID to every pixel. Two-stage CWIS approaches, such as MaskRCNN, always include bounding box estimation branch and a FCN- mask segmentation branch, working in a 'detect- then-segment' way. To improve efficiency, one-stage methods such CenterMask Dai al. (2016), YOLACT Bolya et al. (2019) and BlendMask Chen et al. (2020) have been proposed, which remove the proposal generation and feature grouping process. To further free the CWIS from the local box detection, Wang et.al [Wang et al. (2020a) proposed SOLO and obtained on par results to the above methods. In recent years, the methods Fang et al. (202T); I) Dong al. (202I), following DETR Carion it al.] 20200, consider the instance segmentation task as an ensemble prediction problem. In addition. Cheng et al. proposed an universal segmentation framework MaskFormer Cheng et al. (2021) and its upgrade version Fask2Former Cheng et al. (2022), which even outperforms the state-of-the-art architectures specifically designed for the CWIS task.
Notably, two-stage method CenterMask preserves pixel alignment and separates the object simultane- ously by integrating the local and global branch. Although introducing the global information in this way helps improve the mask quality in CWIS, it cannot handle the open-world task very well, because CenterMask multiplies the local shape and the cropped saliency map to form the final mask for each instance. There is no separate loss for the local shape and global saliency. When such a method faces the incomplete annotations in OWIS tasks, the generated mask predictions corresponding to the unlabeled instances would still be punished during training, making it difficult to discover novel objects at inference. The efficient way to jointly take advantages of global and local information in OWIS tasks deserves to be explored.
Open-world instance segmentation OWIS task Wang et al. (2021) here focuses on the following aspects: (1) All instances (without stuff) have to be segmented; (2) Class-agnostic pixel-level results should be predicted with only instance ID and incremental learning ability is unnecessary. Several OWIS works have recently been developed. Yu et al. Duu et al.) (202T) proposed a two- stage segmentation algorithm, which decoupled the segmentation and detection modules during training and testing. This algorithm achieves competitive results on the UVO dataset thanks to the abundant training data and the introduction of effective modules such cascade RPN Vu al. (2019), SimOTA Ge et al. (2021), etc. Another work named LDET Saito et al. (021) attempts to solve the instance-level incomplete annotation problem. Specifically, LDET first generates the background of the synthesized image by taking a small piece of background in the original image and enlarging it the same size the original image. The instance is then matted the foreground of the synthesized image. The synthesized data is used only to train the mask prediction branch, and the rest of the branches are still trained with the original data. Meanwhile, Wang et al. proposed GGN Wang et al. (2022). an algorithm that combines top-down and bottom-up segmentation ideas to improve prediction accuracy by generating high- high-quality pseudo-labels. Specifically, Pairwise Affinity (PA) predictor is trained first, and grouping module is used to extract and rank segments from predicted PA to generate pseudo-labels, which would be fused with groundtruth to train the segmentation model.
3 METHODOLOGY
In this section, we first define the OWIS problem with both fully and semi-supervised learning. Then the architecture of our TOIS and the proposed cross-task consistency loss are introduced in Section
3

---

Under review as a conference paper at ICLR 2023
Objectness prediction branch
Only conpeted with saaple
Objectness
Decodur
Transfoenter
MLP
Coupert: volk biczed and uakivted sxcples
Clam-sgsentic madio
Oroundrch
Backbone
Decadeer
Pixal
Mask
Aye
Mask prediction branch
Eeregmmnt iigine
Cross tasss-task
consistency loss CAN.UONN ....
Foreground
Foreground prediction branch
4
Figure Overall framework of the proposed TOIS. The mask prediction branch generates the predicted masks, while the objectness prediction branch computes the objectness score for each mask. The foreground prediction branch segments a foreground region to guide the optimization other two branches
3.3 and 3.4, respectively. Finally, Section 3.4 and 3.5 show how to optimize the TOIS fully and semi-supervised way, respectively.
3.1 PROBLEM DEFINITION OF OWIS
The open-wer spendworld instance segmentation (OWIS) aims to segment all the object instances (things) of any class including those that did not appear in the training phase. Technically, OWIS task to produce set of binary masks, where each mask corresponds to class-agnostic instance. The pixel value of in the mask indicates part of an object instance while 0 indicates not.
3.2 MODEL ARCHITECTURE
Our proposed TOIS framework consists of three branches to alleviate the incomplete annotating, as shown in Figure 2 Basically, we follow the design one-stage Fask2Former[hhemm et al. (2022). The objectness prediction branch estimates the weighting score for each mask by applying sequential Transformer decoder and MLP. The mask prediction branch predicts the binary mask for each instance. It first generates N binary masks with N ideally larger than the actual instance number K₁, which is the number of annotated object instances in given image i. Each mask is multiplied by weighting score with value between and indicating if mask should be selected as an instance mask. This process generates the mask in an end to end way, which avoids to miss the instance because of poor detection bounding boxes and meanwhile reduces the redundant segmentation cost for each proposal. We refer to Cheng et al. (2021 2022 for more details.
The foreground prediction branch is light- fully convolutional network estimate the foreground regions that belong to any object instance. The more detailed design of the foreground prediction branch is in the Appendix. This guides the training of the mask branch through our cross-task consistency loss proposed in the following Sec. 3.3 Once training is done, we discard this branch and only use the objectness and mask prediction branch at inference time. Therefore, we would not introduce any additional parameter computational redundancy, which benefits the running efficiency.
3.3 LEARNING WITH THE CROSS-TASK CONSISTENCY REGULARIZATION
A critical limitation of the OWIS is the never-perfect annotations due the difficulties in annotating class-agnostic object instances. Towards alleviating this issue, we propose a regularization to provide extra supervision guide the OWIS model training under incomplete annotations.
4

---

Under review as a conference paper at ICLR 2023
We construct a branch predict the foreground regions that belong to any of the object instances. Formally we create the the foreground annotation G(x,z) y) calculated by
G( v) - { 0. if g = 0
(1)
1, otherwise,
where (x, y) is one of the annotated object instances for the current image and the union of g defines the foreground object regions. Here (I,, y) denotes coordinate of a pixel in the an image. We use (zx, y) as labels to train the foreground prediction branch.
Our consistency loss encourages the model outputs have the relationship indicated in Eq which states that the the foreground prediction should be the union of instance predictions. To do so, we use the following equation an estimate of the foreground from the instance prediction:
K Σ
G(z, y) = +
m2 (z,
(2)
where m² means the confidence of pixels in j-th predicted mask, and & represents the Sigmoid function. Then, let the foreground prediction from the foreground prediction branch be F. our cross-task consistency loss to make and G(z, y) consistent, which finally leads to the following loss function.
Le DICE + (C,
where DICE and BCE denote the dice-coefficient loss Milletari et al. (016) and binary cross-entropy loss, respectively.
Consistency loss enjoys the following appealing prop- erties. self-calibrated and independent with the
incompleteness level of labels. As shown in Figure 3 for a instance mistakenly annotated as background, but the foreground prediction branch and mask prediction branch both correctly find it, the model would be pun- ished through mask loss and foreground loss. However, the consistency loss thinks this prediction is correct. In this way, consistency loss down-weights the adverse effects caused by other unreliable segmentation losses. The mitigation and the compensation factor synergize to relieve the overwhelming punishments on unlabeled instances.
Maaad veutrios
2OIS
Lytba
Figure 3: Working principle of consistency loss.
3.4 FULLLY-SUPERVISED LEARNING
The overall supev-sue optimization of the proposed TOIS is carried out by minimizing the following joint loss formulation by.
Ly aLm + BLp + 7he + wLor
where
Lm = BCE + DICE(m.g)
(5)
Lp (C, + DICE
(6)
L. = BCB(..)):
where Im, Ipp and L., denote the loss terms for mask prediction, foreground prediction, and objectness scoring, respectively. a, 33, 7 and w are the weights the corresponding losses. m and g represent the predicted masks and corresponding groundtruth, respectively. F and G is the foreground prediction result and the generated foreground groundtruth, while the estimated objectness score is denoted with is set of binary values that indicate whether each mask is an instance. Before computing the Lms matching between the set of predicted masks and groundtruth has been done via the bipartite matching algorithm defined in Cheng al. (022).
5

---

Under review as a conference paper at ICLR 2023
3.5 EXTENSION TO REMI-SUPERVISER LEARNING
Due to the ambiguity of the instance definition in OWIS, it is much harder for the annotators to follow the annotation instruction, and this could make the annotations for OWIS expensive. It is desirable if we use unlabeled data help train OWIS models. In this regard, our proposed cross-task consistency loss only requires the outputs of both predictors to have a consistent relationship indicated in [ and does not always need ground truth annotations. Thus, we apply this loss unlabeled data, which becomes semi remi-supervised learning. Specifically, the easier-to-leam foreground prediction branch is able to learn well through few labeled images in the warm-up stage. Then the resulted foreground map can serve as constraint to optimize the open-world mask predictions with the help of our cross-task consistency loss, when the labels do not exist. In this way, our Semi-TOIS achieves good trade-off between the annotation cost and model accuracy.
Semi-supervised learning process. Given a labeled set Dj = [2,, 9.) Ny and an unlabeled set Du - | our goal is to train an OWIS model by leveraging both a large amount of unlabeled data and smaller set of labeled data. Specifically, we initially use D; to train the TOIS as warm-up stage, giving good initialization for the model. We then jointly train the OWIS model on the both labeled and unlabeled data. For the labeled data, we employ the loss function defined in Eq For the unlabeled data, we apply only the cross-task consistency loss Tu..
4 EXPERIMENTS
For demonstrating the effectiveness of our proposed TOIS, we compared it with other supry-su methods through intra-dataset and cross-datase evaluations. We also performed ablation studies in these two settings to show the effect of each component. Moreover, apply the proposed cross-task consistency loss for semi-supervised learning and test our method on the UVO validation set.
4.1 IMPLEMENTATION DETAILS AND EVALUATION METRICS
Implementation details Detectron2 Wu et al. (2019) is used to implement the proposed TOIS framework, multi-scale feature maps are extracted from the ResNet-50 He et al. (2016) or Swin Transformer Liu et al. (20TT) model pre-trained on ImageNet Dene et al. (2009). Our transformer encoder-decoder design follows the same architecture as Mask2Former Cheng et al. (2022). The number of object queries M is set to 100. Both the ResNet and Swin backbones use an initial learning rate of 0.0001 and weight decay of 0.05. A simple data augmentation method, Cutout Deeries & Taylor (2017), is applied to the training data. All the experiments have been done on 8 NVIDIA V100 GPU cards with 32G memory.
Pseudo-labeling for COCO train set Pseudo-labeling is common way to handle incomplete annotations. To explore the compatibility of our method and the pseudo-labeling operation, we employ a simple strategy to generate pseudo-labels for unannotated instances in the COCO train set Lin et al. (2014) in our experiments. Specifically, we follow typical self-training framework, introducing the teacher model and student model framework generate pseudo-labels. These two models have the same architecture, as shown in Figure 2 but are different in model weights. The weights of the student model are optimized by the common proppaaopa,,tion while the weight of the teacher model is updated by computing the exponential moving averages (EMA) of the student model. During training, the image i is first fed into the teacher model to generate some mask predictions. The prediction whose confidence is higher than certain value would be taken as pseudo-proposse The state Sij of the pseudo-proposal Pij is determined according to Equation (8)
Si = True, argmax & E
(8)
False, otherwise,
in which 9r means any ground truth instance in the image i. y denotes the IOU calculating function,
and is a threshold to further filter the unreliable pseudo-proposals. Finally, pscudo-proposale with states True would be considered as reliable pseudo-labels. Here, the confidence and IOU threshold e set to 0.8 and 0.2, respectively. Then, we jointly use the ground truth and the pseudo-labed to form the training data annotations. a region is identified as belonging to an instance in the lseudo-label, it will be considered as a positive sample during training.
for selecting pseudo-labels

---

Under review as a conference paper at ICLR 2023
Table 1: Results of UVO-train UVO-val intra-dataset evaluation.
Metric
Backbone
AP (0n(%)
AP,(%)
m
AP)(%)
AR (00
AR
MaskRCNN
R-50
13.41
4,91
12.33
17.45
22.77
20.01
LDET
R-50
16.25
3.27
13.58
22.93
35.64
23.73
Farmer
R-50
21.85
6.16
16.82
31.65
41.18
28.26
TOIS (Ours)
R-50
23.38
6.59
17.35
34.23
29.24
Former
Swin-B
33.27
9.34
25.21
47.80
50.81
37.49
TOIS (Ours)
Swin-B
38.02
12.31
28.64
53.22
54.74
41.78
Evaluation metrics
The Mean Average Recall (AR) and Mean Average Precision (AP) Lin et al. (2014) utilized to measure the performance of approaches in class-agnostic way.
4.2 FULLY-SUPERVISED EXPERIMENTAL SETTING
Intra-dataset evaluation UVO is the largest open-world world instance segmentation dataset. Its training and test images from the same do- main, while they do not have any overlap. Here, we perform the learning process of TOIS on the UVO-train subset and conduct the test experi- ments on the UVO-val subset. Besides, we split the COCO dataset into 20 seen (VOC) classes and unseen (none-VOC) VOC) classes. We train a model only on the annotation of 20 VOC classes and test it on the 60 none- VOC class, evaluating its ability of discovering novel objects.
Table Results of COCO2017-train(VCC) COCO2017-val(monte (-ll((olonnne- intra-set evaluation. 8,m and denote small, middle and large size of instances.
A5,
Mast2Toree Fommer
9.21
4.56
8.79
19.30
TOIS
11.03
4.87
9.24
26.11
Cross-dataset evaluation Open-world setting assumes that the instance can novel classes target domain. Therefore, it is essential for the OWIS method to handle the potential domain gap with excellent generalization ability. Cross-dataset evaluation, in which training and test data come from different domains, is necessary to be conducted. Here, first train the proposed TOIS model and compare methods on the COCO-train subset, while testing them on the UVO-val dataset to evaluate their generalizability. Then extend the experiments models on the Cityscapes Cordts et al.) (2016) dataset and evaluating them on the Mapillary Neuholdd al. (2017). Cityscapes have 8 foreground classes, while Mapillary contains 35 foreground classes including vehicles, animals, trash can, mailbox, etc.
autonomous driving scenario, training the
4.3 FULLY-SUPERVISEE EXPERIMENTAL RESULTS
Intra-dataset evaluation The re- sults are illustrated in Table [ The single-stage approaches based the mask classification framework perform better than other two-stage methods. Among them, our pro- posed TOIS achieves significant performance improvement over the Mask2Former baseline, which is 4.75% in AP and 3,93% in Aft when using the Swin-B backbone. For VOC-none--V VOC setting, the ex- perimental results are shown in Table 2 which verified that our proposed method can improve the performance for all instances, especially large ones.
Table 3: Results of COCO2017-tran UVO-val cross-
dataset evaluation.
AR
AP.a.(5)
AP.(S)
AP.(S))
AP:
B.MinttC S Celse es 1000
20.40
RCvd RCnd
20.87
Poindem
R:50
37.02
20.40
34.57
R-50
19:05
28.05
LDE LDEE Sain b 200
R-50
42.63
21.27
5.66
17.52
18.38
ocN Warg
R-50
43.30
20.30
$.70
18.20
27.30
Maazzriing Lanzz M02
R30
48.71
20.31
6.46
16.29
40.37
TOIS (Quss)
R.50
$1.26
27.02
7,50
18.61
43.42
SWi B
31.35
25.15
1.29
19.31
15.03
TOiSSOurs)
Swis-B
54.86
32.21
9.83
21.92
$8.69
evaluation For the COCO->UVO task, according to Table 3 it is clear that the proposed TOIS outperforms all previous methods, achieving a new state-of-the-as Af at 54.86% which is 11 1.56% higher than previous state-of-the-art method GGN Wang et al. (2022). We also applied the proposed techniques to another classic one-stage method SOLO V2 Wang et al. (020b)) The experimental results in Table show that it improves AR100 and AP by 3. 1%11% and 7.7% compared to SOLO V2. For the Cityscapes-A Mapillary task, the overall AP and AR of TOIS still surpass the performance of other state-of-the-arr methods in Table 5 which demonstrates the
7

---

Under review as a conference paper at ICLR 2023
Table 4: Results of TOIS with SOLOV2 structure ( Table 5: Cross-dataset evaluation autonomous driving scenes. Results of Cityscapes Mapillary.
UVO-train UVO-val).
Method ARR33
MI
LOST
Mask2Fomme
OSISi
8::0
18:11
20.35
7.5
23
7.5
84
V21GI5
1659
42.52
2334
6.71
06.00
5633
61
53
75
(a) Groundtruth
(b) MaskRCNN
(c) LDET
(d) Mask2Former
(c) TOIS (Ours)
Figure Visualization results of COCO cross-dataset evaluation. The predicted boxes of two-stage methods MaskRCNN and LDET are also drawn. Proposed TOIS can discover both unlabeled objects (first row) and unseen class of instances (second row) as shown in red boxes.
effectiveness of our proposed techniques. We show some of the COCO-vVO visualization results in Figure to qualitatively demonstrate the superiority of our method. Please refer to the supplementary material for more qualitative examples.
4.4 ABLATION STUDY
We perform cross-dataset and intra-dataset ablation studies analyze the effectiveness of each component in the proposed TOIS, including the foreground prediction branch and the cross-task consistency loss. We also try combinations of the pseudo-label generation strategy and our cross-tast consistency loss to investigate the individual and synergetic effects of them. Using the SwinB backhone, these models are trained on the COCO-train subset and the UVO-train subset, respectively. The metrics reported in Table aare tested on the UVO-val dataset.
Effectiveness foreground prediction branch Table shows that although a separate fore- ground prediction branch can guide the method to optimize towards the direction of discover- ing foreground pixels, it only slightly boosts the performance.
Table 6: Ablation results of the proposed com- ponents by cross-dataset and intra-dataset evalua- tions. Foreground prediction (FP), Cross-tase con- sistency (CTC) loss, Pseudo label (PL).
Effectiveness of cross-task consistency loss Cross-task consistency loss has positive effect on both sparse annotated (COCO) and dense annotated (UVO) training dataset. The values of A AP00 and AR10 increase significantly 2.74% 1 and 2.49% 1 COCO while 2.90% t and 3.35% on UVO) after applying the cross- task consistency loss well as the foreground prediction branches together. This result outperforms the TOIS counterpart with only pseudo-labeling showing effectiveness. In addition, jointly utilizing cross- task consistency loss well the pseudo-labeling strategy leads to performance improvements on two settings, which demonstrates the synergistic effect of both approaches.
Component
Train COCO
Train UVO
FP CTC luss PL APioo(S)) AR (
AP (b) AR
28.65
51.54
35.12
51.39
29.02
51.60
35.55
51.73
30.09
52.97
32.94
51.64
4
31.39
53.83
38,02
54.74
30.17
52.98
33.35
50.90
32.21
54.86
37.71
52.27
Effectiveness of pseudo-labeling Pscudo-labeling is not always necessary and powerful for any types datasets. As shown Table the AP100 and AR of the COCO trained model increase by 1.35% and 0.78%, respectively, after applying the pseudo-label generation. However, pseudo- labeling causes a performance degradation (e.g. 281% I in AP to a model trained the UVO dataset. Compared with COCO, the UVO dataset is annotated more densely. We conjecture that the
8

---

Under review as a conference paper at ICLR 2023
Table 7: Results of our TOIS and classic semi- supervised method vVO
Table 8: Results of our TOIS and recent end to end method on UVO-val.
Trinin
UVO unoizion
usstation Faly-TOss
2007
21.77
1500
1031
1030
22:15
25-2
ARoe(9)
40.09
4082
41.36
45,42
25:08
58.06
41.44
47.66
background annotations of UVO are more reliable than those of COCO, where carefully selected pseudo-labels are more likely represent unlabeled objects. The generated pseudo-labels of UVO contain higher noises than those of COCO. These additional noisy labels mislead the model training.
4.5 SEMI-SUPERVISED LEARNING EXPERIMENT
Experimental setting We have divided the UVO-trat train dataset into the labeled subset D; and the unlabeled subset D.. Semi-supervised model Semi-TOIs is optimized as described in Section 3.5 on Dj Dy. while the superviee method Labelled- Ooaedd-Oll--OOO is trained merely on the Dz. To ensure the comprchensiveness of the experiments, two different data division settings included in our experiments: (D D2=30%, D==70%) ] and (Dz=50%, D,=50%). }. The backbone applied here is Swin-B. We also implemented the classic Mean teacher model and simple pseudo-label method based on the Mask2Former to perform comparison. Note that our semi-supervised setting is different from that in some previous work, like MaskXRCNNHu (2019). They assume that the training set C' = A U B, where examples from the categories in A have masks, while those in have only bounding boxes.
al. (2018) and Shapemask al.)
Results and analyss As presented in Figure the Semi-TOISso model trained on the UVO with 50% annotated data outperforms the Semi-TOIS model learning with 30% labeled training images. However, the performance increase between the Semi-TOTS and Semi-TOT S50 is slight. In addition, Semi-TOTS improves Saahelled-Only-TOOee in AP and Af. respectively. Compared to Sabelled-Only-TO1155 Semi-TO1Sto still achieves significant advantages (2.36% in AP100 and 6.12% in AR100). These results reflect that cross-task consis- tency loss has the ability to extract information from unlabeled data and facilitates model optimization in the semi-supervised setting. It is notable that the results of Semi-TO1S30 are even better than those of Labelled- Only-TO1Sso- This illustrates that the information dug out by the cross-task consistency loss from the remaining 70% unlabeled data is more abundant than that included in 20% fully-labeled data. Therefore, our algorithm can achieve better performance with fewer annotations. This characteristic is promising in solving the OWIS problem. In addition, we also compared the semi-TOIS with classic semi-supervised method and recent end to end segmentation method. The results in Table 7 and show our advantages over the compared methods.
55-
TOIS
50000000
50
Semi TOISg
-
Semi TOSS
45
by 3.36% and 5.33%
40
1015TOIS
TOSS TOIS
TOIS
35
35
AR100
30-
Semi TOIS Seni-TD18p
25
+3.5
TOi L0g-
20
40
60
80
100
Rutio of unlabeled training data on UVO-trinnig set
Figure 5: Comparison between TOIS,
Labelled-OOly-TOISSLOO---- TOIS.
and semi-
5 CONCLUSION
This paper proposes the first transformer-based framework (TOIS) for the open-world instance segmentation task. Apart from predicting the instance mask and objectness score, our framework introduces a foreground prediction branch to segment the regions belonging to any instance. Utilizing the outputs of this branch, we propose a novel cross-task consistency loss to enforce the foreground prediction to be consistent with the prediction of the instance masks. We experimentally demonstrate that this mechanism alleviates the problem of incomplete annotation, which is critical issue for open-worte open-world segmentation. Our extensive experiments demonstrate that TOIS outperforms state-of- the-art methods by a large margin on typical datasets. We further demonstrate that our cross-tar consistency loss utilize unlabeled images to obtain some performance gains for semi-supervised instance segmentation. This is an important step toward reducing laborious and expensive human annotation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference computer viston, pp. 9157-9166. 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan. Blendmask: Top-down meets bottom-up for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8573-8581, 2020.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022.
Tianheng Cheng, Xinggang Wang, Lichao Huang, and Wenyu Liu. Boundary-preserving mask r-cnn. pp. 660-676, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1150-3158, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708. 04552, 2017.
Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects by learning queries. Advances in Neural Information Processing Systems, 34, 2021.
Yuming Du, Wen Guo, Yang Xiao, and Vincent Lepetit. 1st place solution for the uvo challenge on video-based open- open-world segmentation 2021. arXiv preprint arXiv:21 16.11 2021.
Yuxin Fang, Shusheng Yang, Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6910 9919-6919, 2021.
Zheng Ge. Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint 100 100.00000, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. pp. 4233 4241, 2018.
Lei Ke, YunWing Tai, and Chi-Keung Tang. Deep occlusion-aware instance segmentation with overlapping bilayers. 2021.
10

---

Under review as a conference paper at ICLR 2023
Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. 2022.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. pp. 9799-9808, 2020.
Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung- Yi Lin. Shapemask: Learning to segment novel objects by refining shape priors. pp. 9207-9216, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on vision (3DV), pp. 565-571. IEEE, 2016.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp. 4990-4999, 2017.
Kuniaki Saito, Ping Hu, Trevor Darrell, and Kate Saenko. Learning to detect every thing in an open world. arXiv preprint arXiv:2112.0166 2021.
Thang Vu, Hyunjun Jang, Trung Pham, and Chang Yoo. Cascade rpn: Delving into high-quality
region proposal network with adaptive convolution. Advances in neural information processing systems, 32, 2019.
Weiyao Wang. Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10776-10785, 2021.
Weiyao Wang, Matt Feiszli, Heng Wang, Jitendra Malik, and Du Tran. Open-world instance segmentation: Exploiting pseudo ground truth from learned pairwise affinity. CVPR, 2022.
Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In European Conference Computer Vision, pp. 649-665. Springer, 2020a.
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Yuxin Wu. Alexander Kirillov, Francisco Massa, Wan-Yer Lo, and Ross. Girshick. Detectron2. https://github. comffueebokrrearckddeecono 2019.
A
APPENDIX
Please refer to the additional supplement material.
11


Reviews:
[{"review_id": "yj36QM2EhaZ", "reviewer": "Reviewer_x3r9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the task of open-world instance segmentation in images (no predefined set of classes). The proposed method extends on an existing state-of-the-art single-stage (without employing a dedicated proposals procedure) approach, such as Mask2former, through the addition of a foreground segmentation branch. The foreground prediction map is used as an auxiliary supervisory signal for the instance prediction branch to increase the consistency among these maps. The consistency loss applied on top of these predictions also alleviates the problem of incomplete instance annotation which could be considered a core contribution of this work (working with labeled and unlabeled data simultaneously - semi-supervised scenario).", "summary_of_the_review": "Besides the aforementioned issues with the current submission, I do not find it fit for publication not just for the structure or lack of clarity, but mostly due to its incremental contribution to existing works. ", "strength_and_weaknesses": "Strengths:\n* The tackled problem is challenging and of interest to the research community - open-world instance segmentation is far from being solved - since it implies the discovery of novel object instances that were not seen during training\n\nWeaknesses:\n* Problem statement description could be improved. The introductory section does not properly address the tackled task or the way in which this work contributes to current issues. \n* Hard to grasp from the paper what are the authors' contributions - I did not find the paper an easy read.\n* The authors rename the Mask2former architecture as SOIS and wrongfully consider it their contribution since the architecture itself did not change as the authors stated themselves but rather the scenario in which it was used (extension to a semi-supervised scenario by employing a proxy constraint, in the form of a consistency loss between predicted masks of two existing branches in the architecture)\n\nOther comments:\n* Benchmarks are not referenced when first mentioned - same goes for COCO and UVO in the abstract - at least mention that these are datasets. \n* Figure 1 is crowded and hard to follow all of its parts. It would make more sense to split it into two parts - a, b, c - Part 1 and d, e, f, g - Part 2. In (b) consistency is misspelled and Cityscapes is misspelled in the figure caption.\n* Figure 1c, 1d, 1e and 1f are not referenced in the paper.\n* Figure 2 is not informative at all - What is the purpose of that backbone? - What is the format of the output? What is the input of the FCN in the foreground prediction branch and why not use the RGB image? The image caption does not complement the figure - most of the notations are not explained and the flow of the different branches is very hard to follow. Also, sentences such as \"the foreground prediction branch segments a foreground region\" can be derived from the naming of these components. Authors should highlight only the information that cannot be inferred solely from the naming.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "c1HHFLmrbt", "reviewer": "Reviewer_THoq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an open world instance segmentation framework, which contains three stages: 1) it proposes an auxiliary task of predicting total foreground on the basis of existing query-based mask2former model. 2) it includes a cross-task consistency loss and improve results by correcting outputs between two tasks. 3) the paper also designs a way of adding pseudo labels on COCO, which gain some improvement on open-world settings, including semi-supervised setting.", "summary_of_the_review": "Limited technical novelty. Deeper theoretical analysis is desired. My rating will be adjusted according to the author response.", "strength_and_weaknesses": "Pros:\n\n1. The paper proposes a single-stage open-world Instance segmentation framework based on Mask2Former.\n2. Adding an auxiliary task and a consistency loss for self-supervised learning, which improves a lot on both UVO-to-UVO and COCO-to-UVO settings.\n3. Experiments demonstrate the semi-supervised framework can achieve remarkable performance with much smaller amount of labeled data.\n\nCons:\n1. Although the paper proposed a new constraint called “cross-task consistency loss” for OWIS problem, it lacks of sufficient validation by either theoretical interpretation or experimental evidence how the cross-task consistency loss works for providing an error-correcting mechanism for handling noisy annotations. How does the “cross-task consistency loss” help to handle and separate neighboring occlusion objects?\n\n2. The tech novelty of the paper is weak and incremental. Being first to adapt Mask2Former to OWIS is an incremental idea. Also, it should be called query-based or transformer-based, instead of one-stage. Can the author give proof on why the auxiliary task of predicting total foreground mask will help novel object detection?\n\n3. Missing comparison/discussion with existing semi-supervised instance segmentation methods, such as [a, b].\n\n[a] Learning to segment every thing. CVPR, 2018.\n\n[b] Shapemask: Learning to segment novel objects by refining shape priors. ICCV, 2019.\n\n4. Missing two-stage works on Closed-world instance segmentation, such as [c, d, e].\n\n[c] PointRend: Image Segmentation as Rendering. CVPR, 2019.\n\n[d] Deep occlusion-aware instance segmentation with overlapping bilayers. CVPR, 2021.\n\n[e] Boundary-preserving Mask R-CNN. ECCV, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PyJXtfPEBFg", "reviewer": "Reviewer_baa2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for single-stage open-world instance segmentation. It is the first to uses the SOTA Mask2Former architecture for this task. Additionally, to tackle the problem of \"missing annotations\" while training for open-world segmentations with the limited category labels in the training dataset, the authors propose a foreground prediction branch along with a cross-task consistency loss. The authors further show that since the latter loss does not require any labels, it can also be effectively employed  to train networks in a semi-supervised manner. The authors show quantitatively the superior accuracy of their approach in comparison to the existing state-of-the-art approaches on Open-World instance segmentation, on several baseline datasets. Additionally, they also show the contribution of each component of their design via ablation studies and the usefulness of their proposed approach in semi-supervised learning.", "summary_of_the_review": "Overall the paper makes a significant contribution in advancing the field of open-world instance segmentation. Hence I am leaning towards accepting this paper, with the caveat that the authors improve the claims and clarity of their paper as noted in the \"weaknesses\" section above.", "strength_and_weaknesses": "Strengths:\nNovelty: This paper tackles the problem of open-world instance segmentation, which is a largely open, albeit an important problem in computer vision research. If solved it has huge implications in being able to go from successfully segmenting a few hundred categories of objects today to potentially infinite ones in the future. Hence, it is important to further develop this line of research. \n\nIn terms of the proposed method itself, there are several novel aspects to it. The authors are the first to apply the SOTA transformer-based Mask2former architecture to the task of open-world instance segmentation. Given the known success of the Mask2former in the closed-world setting it is not surprising that also helps to significantly boost accuracies for the proposed method in the open-world setting. Hence, I consider this as a somewhat incremental contribution. Nevertheless, it is to the authors' credit that they are the first to numerically show this to be true. \n\nThe more interesting aspects of the authors' method, however, are the proposed foreground prediction branch and the self-supervised cross-task loss associated with it, which are designed to deal with the problem of \"missing annotations\" in the COCO dataset. I found this design to be clever, task-specific, conceptually correct and numerically shown to help in boosting the accuracy for open-world instance segmentation.\n\nSignificance of results:\nIn absolute terms the proposed method helps to significantly improve the accuracy of the open-world instance segmentation task. That said, the major boost comes from using Mask2Former and less from the other components. Nevertheless, this work is significant in advancing accuracy of this task. Additionally the proposed background prediction module is shown to be multi-purpose. It can be used both for open-world segmentation and for semi-supervised training.\n\nWeaknesses\nClaims:\n1. The authors claim in the introduction \"We propose a Singe-stage Open-world Instance Segmentation (SOIS) for the first time\". This is technically not true. The previous GGN work is also a single-stage method based on SOLOv2. In relation to the existing work, the novel contribution of this work is to be the first to apply a transformer-based Mask2Former architecture to the task of open-world instance segmentation. The authors should correct their claims appropriately.\n\nExperiments:\n2. How sensitive is the proposed method's success to choosing the appropriate weight for the $L_c$ loss term to negate the effect of the network incorrectly penalizing missing detections? Does it need to be set differently for the different datasets?\n\n3. In Table 1, what is the intuition behind the proposed foreground prediction loss helping in the intra-dataset setting? If it is mostly designed to catch un-annotated detections, how does it help to segment the labeled annotations?\n\n4. Section 4.5. \nIt would be better to call the Fully-SOIS to \"Labelled-Only-SOIS\". The term \"full\" seems to imply that the full training set of OVU was used versus just the labeled 30%/50% set. Related to this point, in figure 5, for completeness the authors should also show the results of training with the full OVU training set, ie., the numbers from Table 6.\n\nMinor points:\n1. Introduction, Para 2: \"which detects\" --> \"which detect\"\n2. In Fig.2, the masks don't correspond to the input image at all. Although they are meant to show the concept, nevertheless it would be better to at least show masks that are somewhat representative of the input scene and not those of faces!\n3. In several places, \"leaning\" --> \"learning\"\n4. Table 2, clarify what $s$, $m$ and $l$ mean.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FVrvh5CMFn2", "reviewer": "Reviewer_2Nw5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work leverage the Mask2Former for single-stage instance segmentation plus proposes cross-task consistency regularization to achieve the class-agnostic segmentation of all objects in an image. Specifically, the proposed method uses a foreground prediction branch, similar to the idea of saliency prediction, to help the model learn the objects in an image, then encourage the sum of the predicted objects' map to resemble the foreground map. The proposed approach effectively alleviates the incomplete annotation problem in common datasets, furthermore, it is proven to outperform SOTA models.", "summary_of_the_review": "The idea of encouraging the consistency between foreground prediction and instance mask prediction is good, it helps address the long-existing problems in the benchmark datasets and even helps alleviate the annotation burden in the future. Also, the experiments show the proposed method is effective and help boost the performance over the state-of-the-art. However, I am unsure if the consistency loss framework is significant enough to outstand peers.", "strength_and_weaknesses": "Strength:\n1. This paper is easy to follow since the core idea is presented clearly.\n2. The method can help address the missing annotation problems in the public benchmark datasets and further boost the accuracy of SOTA. \n3. The method can enhance the model performance without introducing further model parameters.\n4. The method is compatible with utilizing unlabeled images for semi-supervised learning.\n\nWeakness:\n1. Since the single-stage instance segmentation is from the Mask2Former, the core contribution seems to become cross-task consistency regularization loss term only.\n2. There are a few typos, replicated citations, and redundant spacing in the context.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject