Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the task of open-world instance segmentation in images (no predefined set of classes). The proposed method extends on an existing state-of-the-art single-stage (without employing a dedicated proposals procedure) approach, such as Mask2former, through the addition of a foreground segmentation branch. The foreground prediction map is used as an auxiliary supervisory signal for the instance prediction branch to increase the consistency among these maps. The consistency loss applied on top of these predictions also alleviates the problem of incomplete instance annotation which could be considered a core contribution of this work (working with labeled and unlabeled data simultaneously - semi-supervised scenario).
strength_and_weaknesses: Strengths:
* The tackled problem is challenging and of interest to the research community - open-world instance segmentation is far from being solved - since it implies the discovery of novel object instances that were not seen during training

Weaknesses:
* Problem statement description could be improved. The introductory section does not properly address the tackled task or the way in which this work contributes to current issues. 
* Hard to grasp from the paper what are the authors' contributions - I did not find the paper an easy read.
* The authors rename the Mask2former architecture as SOIS and wrongfully consider it their contribution since the architecture itself did not change as the authors stated themselves but rather the scenario in which it was used (extension to a semi-supervised scenario by employing a proxy constraint, in the form of a consistency loss between predicted masks of two existing branches in the architecture)

Other comments:
* Benchmarks are not referenced when first mentioned - same goes for COCO and UVO in the abstract - at least mention that these are datasets. 
* Figure 1 is crowded and hard to follow all of its parts. It would make more sense to split it into two parts - a, b, c - Part 1 and d, e, f, g - Part 2. In (b) consistency is misspelled and Cityscapes is misspelled in the figure caption.
* Figure 1c, 1d, 1e and 1f are not referenced in the paper.
* Figure 2 is not informative at all - What is the purpose of that backbone? - What is the format of the output? What is the input of the FCN in the foreground prediction branch and why not use the RGB image? The image caption does not complement the figure - most of the notations are not explained and the flow of the different branches is very hard to follow. Also, sentences such as "the foreground prediction branch segments a foreground region" can be derived from the naming of these components. Authors should highlight only the information that cannot be inferred solely from the naming.
clarity,_quality,_novelty_and_reproducibility: Clarity - Could be improved. The figure and caption do not complement each other. Also, there are notations in the paper that are not clearly defined (Section 3.2) What is Ki? The number of annotated object instances in a given image i? Should be explicitly specified.

Quality - The paper could be improved in terms of language and structure - there are also a number of typos and missing references that should be addressed. Most parts of Figure 1 are not referenced in the paper.

Novelty - I do not consider the contribution novel, but an extension of an existing state-of-the-art that was applied in a different learning setup that deals with missing annotations - not the fully-supervised case. For this purpose alone the authors apply a consistency loss computed as a segmentation loss between the instance segmentation branch and a foreground prediction branch and use this as a constraint in learning when labels are missing. 

Reproducibility - The authors have released the code in their supplementary material - the experiments should be easy to reproduce. 
summary_of_the_review: Besides the aforementioned issues with the current submission, I do not find it fit for publication not just for the structure or lack of clarity, but mostly due to its incremental contribution to existing works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work leverage the Mask2Former for single-stage instance segmentation plus proposes cross-task consistency regularization to achieve the class-agnostic segmentation of all objects in an image. Specifically, the proposed method uses a foreground prediction branch, similar to the idea of saliency prediction, to help the model learn the objects in an image, then encourage the sum of the predicted objects' map to resemble the foreground map. The proposed approach effectively alleviates the incomplete annotation problem in common datasets, furthermore, it is proven to outperform SOTA models.
strength_and_weaknesses: Strength:
1. This paper is easy to follow since the core idea is presented clearly.
2. The method can help address the missing annotation problems in the public benchmark datasets and further boost the accuracy of SOTA. 
3. The method can enhance the model performance without introducing further model parameters.
4. The method is compatible with utilizing unlabeled images for semi-supervised learning.

Weakness:
1. Since the single-stage instance segmentation is from the Mask2Former, the core contribution seems to become cross-task consistency regularization loss term only.
2. There are a few typos, replicated citations, and redundant spacing in the context.

clarity,_quality,_novelty_and_reproducibility: 1. The paper is well written, and along with the figures, it is easy to follow overall.
2. The code will be released if accepted as stated in the paper, and I guess the blind reproducibility on the proposed consistency loss seems doable.  
summary_of_the_review: The idea of encouraging the consistency between foreground prediction and instance mask prediction is good, it helps address the long-existing problems in the benchmark datasets and even helps alleviate the annotation burden in the future. Also, the experiments show the proposed method is effective and help boost the performance over the state-of-the-art. However, I am unsure if the consistency loss framework is significant enough to outstand peers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an open world instance segmentation framework, which contains three stages: 1) it proposes an auxiliary task of predicting total foreground on the basis of existing query-based mask2former model. 2) it includes a cross-task consistency loss and improve results by correcting outputs between two tasks. 3) the paper also designs a way of adding pseudo labels on COCO, which gain some improvement on open-world settings, including semi-supervised setting.
strength_and_weaknesses: Pros:

1. The paper proposes a single-stage open-world Instance segmentation framework based on Mask2Former.
2. Adding an auxiliary task and a consistency loss for self-supervised learning, which improves a lot on both UVO-to-UVO and COCO-to-UVO settings.
3. Experiments demonstrate the semi-supervised framework can achieve remarkable performance with much smaller amount of labeled data.

Cons:
1. Although the paper proposed a new constraint called “cross-task consistency loss” for OWIS problem, it lacks of sufficient validation by either theoretical interpretation or experimental evidence how the cross-task consistency loss works for providing an error-correcting mechanism for handling noisy annotations. How does the “cross-task consistency loss” help to handle and separate neighboring occlusion objects?

2. The tech novelty of the paper is weak and incremental. Being first to adapt Mask2Former to OWIS is an incremental idea. Also, it should be called query-based or transformer-based, instead of one-stage. Can the author give proof on why the auxiliary task of predicting total foreground mask will help novel object detection?

3. Missing comparison/discussion with existing semi-supervised instance segmentation methods, such as [a, b].

[a] Learning to segment every thing. CVPR, 2018.

[b] Shapemask: Learning to segment novel objects by refining shape priors. ICCV, 2019.

4. Missing two-stage works on Closed-world instance segmentation, such as [c, d, e].

[c] PointRend: Image Segmentation as Rendering. CVPR, 2019.

[d] Deep occlusion-aware instance segmentation with overlapping bilayers. CVPR, 2021.

[e] Boundary-preserving Mask R-CNN. ECCV, 2020.
clarity,_quality,_novelty_and_reproducibility: Good paper structure. Figure 5 is blurry.
summary_of_the_review: Limited technical novelty. Deeper theoretical analysis is desired. My rating will be adjusted according to the author response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for single-stage open-world instance segmentation. It is the first to uses the SOTA Mask2Former architecture for this task. Additionally, to tackle the problem of "missing annotations" while training for open-world segmentations with the limited category labels in the training dataset, the authors propose a foreground prediction branch along with a cross-task consistency loss. The authors further show that since the latter loss does not require any labels, it can also be effectively employed  to train networks in a semi-supervised manner. The authors show quantitatively the superior accuracy of their approach in comparison to the existing state-of-the-art approaches on Open-World instance segmentation, on several baseline datasets. Additionally, they also show the contribution of each component of their design via ablation studies and the usefulness of their proposed approach in semi-supervised learning.
strength_and_weaknesses: Strengths:
Novelty: This paper tackles the problem of open-world instance segmentation, which is a largely open, albeit an important problem in computer vision research. If solved it has huge implications in being able to go from successfully segmenting a few hundred categories of objects today to potentially infinite ones in the future. Hence, it is important to further develop this line of research. 

In terms of the proposed method itself, there are several novel aspects to it. The authors are the first to apply the SOTA transformer-based Mask2former architecture to the task of open-world instance segmentation. Given the known success of the Mask2former in the closed-world setting it is not surprising that also helps to significantly boost accuracies for the proposed method in the open-world setting. Hence, I consider this as a somewhat incremental contribution. Nevertheless, it is to the authors' credit that they are the first to numerically show this to be true. 

The more interesting aspects of the authors' method, however, are the proposed foreground prediction branch and the self-supervised cross-task loss associated with it, which are designed to deal with the problem of "missing annotations" in the COCO dataset. I found this design to be clever, task-specific, conceptually correct and numerically shown to help in boosting the accuracy for open-world instance segmentation.

Significance of results:
In absolute terms the proposed method helps to significantly improve the accuracy of the open-world instance segmentation task. That said, the major boost comes from using Mask2Former and less from the other components. Nevertheless, this work is significant in advancing accuracy of this task. Additionally the proposed background prediction module is shown to be multi-purpose. It can be used both for open-world segmentation and for semi-supervised training.

Weaknesses
Claims:
1. The authors claim in the introduction "We propose a Singe-stage Open-world Instance Segmentation (SOIS) for the first time". This is technically not true. The previous GGN work is also a single-stage method based on SOLOv2. In relation to the existing work, the novel contribution of this work is to be the first to apply a transformer-based Mask2Former architecture to the task of open-world instance segmentation. The authors should correct their claims appropriately.

Experiments:
2. How sensitive is the proposed method's success to choosing the appropriate weight for the $L_c$ loss term to negate the effect of the network incorrectly penalizing missing detections? Does it need to be set differently for the different datasets?

3. In Table 1, what is the intuition behind the proposed foreground prediction loss helping in the intra-dataset setting? If it is mostly designed to catch un-annotated detections, how does it help to segment the labeled annotations?

4. Section 4.5. 
It would be better to call the Fully-SOIS to "Labelled-Only-SOIS". The term "full" seems to imply that the full training set of OVU was used versus just the labeled 30%/50% set. Related to this point, in figure 5, for completeness the authors should also show the results of training with the full OVU training set, ie., the numbers from Table 6.

Minor points:
1. Introduction, Para 2: "which detects" --> "which detect"
2. In Fig.2, the masks don't correspond to the input image at all. Although they are meant to show the concept, nevertheless it would be better to at least show masks that are somewhat representative of the input scene and not those of faces!
3. In several places, "leaning" --> "learning"
4. Table 2, clarify what $s$, $m$ and $l$ mean.
clarity,_quality,_novelty_and_reproducibility: Clarity: Mostly clear, but could be improved further by addressing the points noted in the weaknesses section above.

Quality: Meets the quality standards of an ICLR submission in terms of the technical contribution and presentation.

Novelty: Sufficiently novel in both the proposed method and the fairly open problem addressed.

Reproducibility: The method is well described. The supplementary provides the implementation details. The authors provide the code and have promised to release it.
summary_of_the_review: Overall the paper makes a significant contribution in advancing the field of open-world instance segmentation. Hence I am leaning towards accepting this paper, with the caveat that the authors improve the claims and clarity of their paper as noted in the "weaknesses" section above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This article addresses the problem of open-world instance segmentation with no predefined classes. The problem is interesting, challenging, and important. The reviewers agree that the article is straightforward, easy to follow, boosts state of the art, does not introduce new parameters, and is compatible with semi-supervised learning.

However, they also agree that the novel contribution is very limiting since it is a straightforward extension of Mask2former where a foreground segmentation branch is added and a cross-task consistency loss. They also highlight that the problem statement and the intro can be improved. They didn't clarify what part of the contribution is from them and what comes from Mask2former. The article overstates the contributions of the method. It also misses comparison with other state-of-the-art articles and other minor comments.

The authors addressed some of the comments, and the paper improved. However, given the limited contribution of the article, I recommend rejecting it.
justification_for_why_not_higher_score: Given the limited contribution of the article, I recommend rejecting it.
Decision: Reject