Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a novel instance segmentation training paradigm that can be combined with any model. Two properties are explored: dataset-level uniqueness and transformation equivariance. This training is combined with quite a few SOTA models, and it is shown that a significant gain of 2-3 AP on COCO. Training is only 5% slower than standard training, with no speed loss at instance. 
strengths_and_weaknesses: Strengths:
+ This training paradigm is novel as far as I can tell, it is properly motivated, clearly explained and both additions are orthogonal, which results in experiment section clearly show
+ When I started reading the paper, I expected that training cost will be much higher than baseline, but upon reading the implementation strategies (namely external memory, sparse sampling and instance-balanced sampling) I was convinced that the loss in speed will be minor, and it was, it is just 5%! Kudos for the engineering aspect of the paper
+ Experimental section is very well designed. Authors compare with a lot of SOTA approaches, they incorporate their method with 4 different representative models. Then, they clearly explain the effect of inter-scene and equivariance, they are both providing performance boost and are orthogonal, so combined they achieve biggest jump. Finally, ablation over hyperparams is very good, it shows that the newly added hyper-parameters are fairly robust (at least on COCO dataset), and it helps the reader select them on their problem.
+ Performance gains are very good, hard to be ignored. Performance of the best result is SOTA. 

Weaknesses:
- Whole paper evaluated on only one dataset. I do know COCO is standard, but for a paper that claims to improve training paradigm, I think maybe one more dataset should be used. Not to achieve SOTA necessarily, but to show benefits on top of a good model. Eg, evaluation on LVIS would greatly improve the paper, in my opinion.
- Minor: When reading Mask2Former paper, it seems their equivalent COCO result is 50.1, not 48.5 like here. Can authors explain in more detail what is the difference in their implementation and why this happens?
questions: I asked my questions in weaknesses. Authors should focus on those in the rebuttal.
limitations: Nothing to add here.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new training framework that exploits query embedding learning. Dataset-level uniqueness and transformation equivariance are introduced and demonstrate promising results on the benchmark datasets.
strengths_and_weaknesses: + Framework contribution: this work brings a new paradigm shift that goes beyond inner-scene training to an inter-scene level query embedding separation.

+ Introducing equivariance is intuitive and the results are convincing.

- The evaluation is somewhat weak. Only COCO is adopted.
questions: In Ln 158, uniqueness and robustness are named as two crucial properties. Can these be consistent with the statement in abstract Ln 5?

If the equivariance is estimated on the feature representation, what makes it different to traditional concepts, e.g. invariance?
limitations: Despite the novelty of introducing equivariance, the in-depth analysis of the mathematical properties is limited. Only empirical results cannot fully reflect the rationale behind the equivariance eqution.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes using two ways to improve the performance of query-based instance segmentation network. First, by using the queries to retrieve the corresponding instance from the whole training dataset, the segmenters are forced to learn instance-uniques queries. Second, by performing geometric transformations and encouraging the network predictions to be equivariant, the authors expect to learn more robust instance-query matching. Experiments show that both ways can improve the instance segmentation effectively. 
strengths_and_weaknesses: Pros:
-The idea is simple and reasonable.

-Overall, the paper is easy to read and understand.

-Experiment results are good. 

Cons:
-The relationship between the proposed two contributions seems not to be clear. And the equivariance-based augmentation seems not to be tailored for instance segmentation, and even not to be tailored for query-based segmentation framework. I think it is a common augmentation trick which may not be suitable to claim it as a contribution of this paper. 

-What is the detailed format of cross-entropy loss and focal loss for $\mathcal{L}_{inter\_mask}$？
questions: See the weakness part.
limitations: The authors didn't discuss the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: - The paper proposes to enhance the dataset-level uniqueness and transformation equivariance of queries in the query-based segmentation methods.
- The proposed method essentially adopts some technics in contrastive learning to learn discriminative queries.
- The method consistently improves baseline methods like CondInst, SOLOv2, SOTA, and Mask2Former.
strengths_and_weaknesses: ## Strengths:
1. The paper exhibits consistent significant improvements over many methods including CondInst, SOLOv2, SOTR, and Mask2Former.
2. The paper reveals a new direction of improvements over query-based instance segmentation methods by enhance the discrimination ability of queries.

## Weaknesses:
1. There are too many equations that makes the paper hard to follow. For example, the notion of $O^I_{n}$ in eq.5 and  $\hat{W^{g}_{n}}$ in eq. 8
2. May also show the iteration time and GPU memory cost in Table 3 (b) and (c) to show the trade-off.
3. K-Net [a] is not mentioned and compared in the paper, which has better speed-accuracy trade-off than SOLOv2 and CondInst.
4. The description is not accurate and should be updated. This paper essentially improves methods that use dynamic kernels, i.e., ‘kernel-based’ methods rather than 'query-based' methods. SOLOv2 [37], CondInst[32], SOTR[39], K-Net [a], MaskFormer[40], Mask2Former [38] they essentially all use some strategy to predict content-aware kernels then use the kernels to perform convolution with the feature map to produce masks. This is explicitly described in the original paper of SOLOv2, CondInst, SOTR, and K-Net, although MaskFormer and Mask2Former describe them as ‘mask embeddings’.
5. The augmentation techniques are not sufficiently studied, i.e., only geometric transformation is studied in Table S2 while photometric transformations and the detailed parameters like cropping range are not studied.
[a] K-Net:Towards Unified Image Segmentation, NeurIPS 2021.
questions: 1. Why color jittering and blur is not suitable? It is quite common in contrastive learning [58,59,60] to use both geometric and photometric transformations, therefore, intuitively the reviewer does not understand why it is not suitable.
2. Methods like MaskFormer, Mask2Former, and K-Net are not only used for instance segmentation but also panoptic segmentation methods, does the proposed method can have a more generic form that works for panoptic segmentation or it can only be used for instance discrimination?
limitations: 1. As asked in the questions, if the methods only applies to instance segmentation, it might be better to claim its scope.
review_title: null

Meta-review: metareview: This paper leverages dataset-level uniqueness and transformation equivariance to improve state-of-the-art instance segmentation methods. 
The reviews were overall positive about the submission: the reviewers especially highlighted the good experimental results, the relevance of the scene level query embedding and its complementarity with the equivariance constraints.  
The authors' feedback brings important answers to some reviewers' concerns. Especially, the new conclusive experiments on LVIS or the extension of the method for panoptic segmentation widens the approach's applicability and has been appreciated. Other answers in the rebuttal did not convinced some reviewers, and there remains issues about the novelty of the approach, the terminology and positioning with respect to 'query-based' approaches, or the extension of the method for photometric equivariance.

The AC carefully read the submission. The AC considers that the idea of querying instances from the whole training dataset is interesting. Despite the limited contribution of the equivariance loss, which has been used in several related scenarios, the design of the whole approach in the context of instance segmentation is relevant and well designed. The experiments are also convincing. It is a pity that the authors did not take the opportunity to update the paper during the discussion period, especially the new experiments and the clarifications requested by the reviewers. Based on the relevance of the approach and its good experimental results obtained over various baselines in several datasets, the AC recommends acceptance. He highly encourages the authors to include the elements discussed in the rebuttal to improve the quality of the final paper.

confidence: Certain
Decision: Accept