Learning Equivariant Segmentation with Instance-Unique Querying
Authors: Wenguan Wang
James Chenhao Liang
Dongfang Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Instance Segmentation
Instance-Unique Querying
Transformation Equivariant Learning

Abstract: Prevalent state-of-the-art instance segmentation methods fall into a query-based scheme, in which instance masks are derived by querying the image feature using a set of instance-aware embeddings. In this work, we devise a new training framework that boosts query-based models through discriminative query embedding learning. It explores two essential properties, namely dataset-level uniqueness and transformation equivariance, of the relation between queries and instances. First, our algorithm uses the queries to retrieve the corresponding instances from the whole training dataset, instead of only searching within individual scenes. As querying instances across scenes is more challenging, the segmenters are forced to learn more discriminative queries for effective instance separation. Second, our algorithm encourages both image (instance) representations and queries to be equivariant against geometric transformations, leading to more robust, instance-query matching. On top of four famous, query-based models (i.e., CondInst, SOLOv2, SOTR, and Mask2Former), our training algorithm provides significant performance gains (e.g., +1.6 – 3.2 AP) on COCO dataset. In addition, our algorithm promotes the performance of SOLOv2 by 2.7 AP, on LVISv1 dataset.

Learning Equivariant Segmentation with Instance-Unique Querying
Wenguan Wang"
James Liang"
ReLER, AAII, University of Technology Sydney
Rochester Institute of Technology
Dongfang Liu
Rochester Institute of Technology
Abstract
Prevalent state-of-the-art instance segmentation methods fall into query-based scheme, which instance masks are derived by querying the image feature using set of instance-aware embeddings. In this work, we devise work that boosts query-based models through discriminative query embedding learning. It explores two essential properties, namely dataset-level uniqueness and transformation equivariance, First, our algorithm uses the queries retrieve the corresponding instances from the whole training dataset, instead of only searching within individual scenes. As querying instances across scenes
training frame-
the relation between queries and instances.
challenging, the segmenters forced learn more discriminative queries for effective instance separation. Second, our algorithm encourages both image (instance) representations and queries be equiv- ariant against geometric transformations, leading to more robust, instance-query matching. On top of four famous, query-based models (i.e., CondInst, SOLOv2, SOTR, and FarkeFormer), our training algorithm provides sigmificant performance gains (e.g..+1.6 A.) on COCO dataset. addition, our algorithm promotes the performance of SOLOv2 by 2.7 AP, on LVISv1 dataset.
Introduction
Instance segmentation, iee,, labeling image pixels with classes and instances, plays a critical role in wide range of applications, e.g., autonomous driving, medical health, and augmented reality. Modern instance segmentation solutions are largely built upon three paradigms: top-down ('detect-then- segment') [1-19], bottom-up Hlabel-then-clusee
[20-28], and single-shot ('directly-predict) [29- 444. Among them, the top-leading algorithms [18, 118,34, 34,33,,99 34-44]4] typically operate a query-based mode, in which set of instance-aware embeddings is learned and used to query the dense image feature for instance mask prediction. The key to their triumph the instance-aware query vectors that are learned to encode the characteristics (e.g.. location, appearance) of instances 43.. By straightforwardly minimizing the differences between the retrieved and groundtruth instance masks, the query- based methods, in essence, learn the query vectors for instance discrimination only within individual scenes.
As result, existing query-based instance segmentation algorithms place a premium on intra-scene analysis during network training. Since the scenario in one single training scene is simple, i.e., the diversity and volume of object instances as well as the complexity of the background are typically limited, learning to distinguish between object instances only within the same training scenes is less challenging, and inevitably hinders the discrimination potential of the learned instance queries.
"authors contributed equally corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

This work brings paradigm shift in training query query-based instance segmenters: it goes beyond the de facto. within-scene training strategy by further considering the cross-scen level query embedding separation of different instances querying instances frrccesfron from the whole training dataset. The underlying rationale intuitive yet powerful: an advanced instance segmenter should be able to differentiate all the instances of the entire dataset, rather than only the ones within single scenes. Concretely, in our training framework, the queries are not only learned to fire on the pixels of their counterpart instances in the current training image, but also forced to mismatch the pixels in other training images. By virtue of intra-and inter-scene instance disambiguation, our framework forces the query-based segmenters to learn more discriminative query vectors capable of uniquely identifying the corresponding ins- tances even at the dataset level. To further facilitate the establishment of robust, one-to-one relation between queries and instances, we complement our training framework with transformation equi- variance constraint, accommodating the equivariance property of the instance segmentation task to geometric transformations. For example, if we crop flip the input image, we expect the image (instance) features and query embeddings to change accordingly, so as to appropriately reflect the variation of instance patterns (e-g-, scale, position, shape, etc) caused by the input transformation.
Exploring intra-and inter-scene instance uniqueness as well as transformation equivariance leads to a gen- eral yet powerful training framework. Our algorithm, in principle, can be seamlessly incorporated into the training process of existing query- bessed instance seg-
50-
00CO Mook AP
Mask2Forerr +ours
menters. For comprehensive evaluation, we apply our algorithm to four representative, query-based models (i.e., CondInst [34]. SOLOv2 [39], SOTR [41]. and Mask2Former [40]) with various backbones (i.e., Res- Net [45], Swin [46]). Experiments on COCO [47] verify our impressive performance, i.e., +2.8 3.1, +2.9 - 3.2. +2.4 - 2.6, and +1.6 2.4 AP gains over CondInst. SOLOv2, SOTR, and Fask2Frrmer, respec- tively (see Fig. 1). Our algorithm also brings remark- able improvement, +2.7 AP, on LVISv [48] dataset, on top of SOLOv2 [39]. These results are particularly impressive considering our training algorithm causes neither architectural change nor extra computational load during model deployment.
Skip
SOTR ours
M Mask2Fomerr(40)
Comdinst
S0L0v2 + ours
40-
Basket 101
Rualessa
SOTR[-1]
Rudiee
Redios
SOLOv20
91
Condinst [34]
ours
model
35-
Bo
100
200
Inference Speed (ms)
Figure Our training algorithm yields solid performance gains over oven -f-tte--ffthe---ee query- based models [34, 39-41] without architec- tural modification and inference speed delay.
2 Related Work
This section summarizes the most relevant research on instance segmentation and equivariant learning.
Instance Segmentation. With the renaissance of connectionism, remarkable progress has been made in instance segmentation. Existing deep learning based solutions can be broadly classified into three paradigms: tow-down, bottom-up. and single-shot. Following the idea of ddtecct-henn "te-t--eegeee'' top- down methods [1 [1-18] predict a bounding box for each object and then ouput an instance mask for each box. Though effective, this type methods complicated and dependent on the priori detection results. In contrast, bottom-up methods [20-27] adopt teebllthhenclusster strategy: learning per-pixel em- beddings and then grouping them into different instances. Albeit simple, this type of methods relies on the performance of post-processing and easily suffers from under-segment or over-segment problems. Inspired by the advance of single-stage object detection [49,50], a few recent efforts approach instance segmentation in single-shot fashion, by coalescing detection and segmentation over pre-defined anchor boxes [29-33], or directly predicting instance masks from feature maps [34-44,51], This type of methods is well recognized and generally demonstrates better speed-accuracy trade-off [S2].
Despite the blossoming of diverse approaches, the vast majority of recent top- hop-performing algori- thms [18, 34439-44] fall into one grand category query-based models. The query- based methods utilize compact, learnable embedding vectors to represent instances of interest and leverage them as queries to decode masks from image features. Their triumph is founded on comprehensively encoding instance-specific properties (e-g., location, appearance) into the query vectors, which significantly increases prediction robustness. For instance, [34,39 39] exploit the technique of dynamic filter [53] to

---

generate instance-specifie descriptors, which are convolved with image feature maps for instance mask decoding. Inspired by DETR [54], [40.42, 43] alternatively leverage Transformer decoder to obtain instance-aware query embeddings and cast instance segmentation as a set prediction problem.
Our contribution is orthogonal and these query-based segmenters can benefit. We scaffold new training framework that sharpens the instance discriminative capability of the query-based segmenters. This is achieved by matching query embeddings with instances within and cross training scenes. Such intra-and inter-scene instance querying strategy is further enhanced by an equívariance regularisation term, addressing not only the uniqueness but also the robustness of instance query relations.
Equivariant Representation Learning. Transformations play a critical role in learning expressive representations by transforming images means to reveal the intrinsic patterns from transformed visual structures [55]. Motivated by the concept of translation equivariance underlying the success of CNNs, numerous efforts (e-g., capsule nets [56,57], group equivariant convolutions [58], and harmonic networks (59]) investigate learning more powerful representations equivariant to generic types of transformations [60, 611. A representation f is said be equivariant with a transformation g for input (say image) / if (((()))=((((()))) in the same manner (or, in a broad sense, predictable manner) given the input transformation g- Many recent self-supervised learning methods [62-64] encourage the representations to be invariant under transformations, i.e., {g((1) fI). As such, invariance can be viewed as a special case of equivariance [65] where the output representation fI) does not vary with the input transformation g.
In other words, the output representation f()) transforms
In our training framework, fully explore the inherent,
wtttiiommmaiioneuquvvviiiaaaaeee nature of the instance segmentation task to pursue reliable, one-to-one correspondence between learnable queries and object instances. This is accomplished by promoting the equivariance of both query embeddings and feature representations with respect to spatial transformations, i.e., cropping or flipping of an input image should result in correspondingly changed feature representation, query embeddings, as well as instance mask predictions. Note that invariance is not suitable for instance segmentation task, it encourages the feature map (and segmentation mask) to not vary with the input transformation. Our algorithm is also in contrast to the common data augmentation strategy, in which the transformed images and annotations are used directly as additional individual training examples, without any constraint about the relation between the representations (and queries) produced from the original and transformed views. Our experimental results (see $4.3) also evidence the superiority of our transformation equivariance learning over transformation-oee based data augmentation.
3 Methodology
Next, we first formulate instance segmentation from a classical view of mask prediction and classifi- cation (53.1). Then we describe our new training framework ($3.2) and implementation details ($3.3).
3.1 Problem Formulation
Instance segmentation seeks partition of an observed image I I HxW33 into K instance regions:
(Y {(M,, <=) where M& (0,
1) (0, Ck (1,...... ,).
(I)
Here the instances of interest are represented by a total of K non-overlap, binary masks {M&} as well as corresponding class labels {c*) (e.g., table, chair, etc). For pixel i its counterpart value in the k-th groundtruth mask, i.e., My (i), denotes whether i belongs to instance k (1) or not (0). Note that the number of instances, K, varies across different images. Existing mainstream (or, more precisely, most top-down and one-shot) solutions approach the task by decomposing the image I into a fixed-size set of soft masks. In this setting, each mask is associated with a probability distribution over all the C categories. The output can thus be represented as a set of N mask-probabilityy pairs:
where M, E xx P.. E
Here A stands for the C-dimensional probability simplex. The size of the prediction set, N, is usually set as constant and much larger than the typical number of object instances in an image. Hence the training objective penalizes the errors of both label prediction and mask estimation:
[((Y,) (Ye)&=.)
Co(nn)) + Lmass Mn, Mo(n));
(3)
3

---

Intee-Scome Instance Discrimination (ours)
delreet
Rintor
By.
Queey creator
h
4Ó Frimt
Intra-Scone Instance
Cinten
icrimination (standard)
derse festurre
f(un. 1i
Fug.
(M
(Mero Fimm
fligping
Transformalis
(l)
Legus
qquivrraacee (ours)
Kog. a (1)))
Eq.
h
(May)
Figure 2: Overview of our new training framework for query- based instance segmentation, Rather than current intra-scene training paradigm, our framework addresses inter-scene instance discrimination and transformation equivariance for discriminative instance query embedding learning (see $3.2). To improve readability, for image we only plot one extra image I' for cross-scene training.
where refers the matching between the prediction and groundtruth sets (established by certain rules 54]). The classification loss Lots is typically the cross-entropy loss or focal loss [66]; the mask prediction loss Lmask can be the cross-entropy loss in [40], dice loss [67] in [34, 391, J, or focal loss in [54]. While many approaches supplement L with various extra losses (e.g., bounding box loss [6, 32, 34, 54, 68], ranking loss [69], semantic segmentation loss [13, 32]), later we will show our cross-scene training scheme is fundamentally different from (yet complementary to) current
scene-wise training paradigm.
3.2 Equivariant Learning with Intra-and Inter-Scene Instance Uniqueness
Query-based Instance Segmentation. As clearly indicated by Eq. 2, the prediction masks {M,, J.Y are the means of separating instances at the pixel level. Current top-performing instance segmen- ters [18, 39-43] typically generate mask predictions in query-based fashion (see the middle part of Fig. 2). Let f be a dense feature extractor D (e.g., an incoder-decoder fully convolutional network [70]) that produces D-dimensional dense embedding I for image i.e., I= f=f(I g HxWxD Then query creator h t is adopted to produce a set of N instance-awe aaware embedding vectors J==1, which are used to query the image representation for instance mask decoding:
(qn E 1dd N
(M.) = (ggn, D) where {qr) (TT)).
Here (.,)) is a certain similarity measure performed pixel-wise, and 1 (typically) refers low- resolution feature representation of image T. Note that position information is integrated to either or both of I and _ to make the model location-sensitive The query creator h is implemented dynamic network [34,39], or Transformer decoder [40,42]. For dynamic network based h, it predicts N convolution filters { J.V dynamically conditioned on the input I and hence (-. refers to convolution (d= D). For Transformer decoder based h, it additionally leverages a set of learnable positional embeddings (omitted for brevity) to gather austance-related context from 1 the collected context stored in {qn J and (.--) is computed as dot product for instance mask decoding (d=D) Eq. 4 informs that, query-basee segmenters in essence learn N compact descriptors {qrm J.W grasp critical characteristics (e.g., appearance and location) of potentially interested instances, and use these instance descriptors queries retrieve corresponding pixels from J. It is thus reasonable assume the discriminative ability of the learned query embeddings is crucial for the performance of qused methods. Viewed in this light, a question naturally arises: how to learn discriminative instance query embeddings? Yet, this fundamental question is largely ignored in the literature so far.
To respond exploit two crucial properties of instance-query matching, namely uniqueness and robustness. This is achieved by addressing dataset-level uniqueness and transformation equivariance during the learning of query- based segmenters, leading powerful training scheme eventually.
Learning with Intra-and Inter-Scene Instance Uniqueness. If we closely scrutinize at the current de facto training regime (cf. Eq. 3) and the work mode of query-based segmenters (cf. Eq. 4), we can find: the mask prediction loss Lmask forces each query qn to match the pixels i / of its counterpart instance k= (n)) ie.. Mn (i) and mismatch the pixels i'e of other instances & (n) i.e.,

---

(+)) withhn image I. That means, the segmenters are only taught to differentiate instances within individual scenes. Evidently. individual scenes are unable to cover diverse instance patterns; the scareity of negative instances an intrinsic limitation of the current scene-wise training regime -inevitably impairs query embedding learning. This analysis motivates us to design a cross-scene training strategy, where the segmenters are forced to learn to query the target instances from the whole training dataset, instead of only matching queries and instance pixels within the same scenes. Thus we take step fürther by simultaneously leveraging the intra-and inter-scene context to promote query embedding learning. Query-based segmenters are encouraged to develop dataset-level, cross-instance discriminativeness and perform reliable mask prediction using instance uniqueness.
Let I be the training set of [Z] images and {q the instance queries generated from image III. During training, rather than only applying {qn) I== for the counterpart image I to get N intra-image instance prediction masks {M, J.y (cf. Eq. 4), we further use {qn J.Y. query other training images I'EI for cross- scene instance disambiguation. This will result in N inter-image instance prediction masks (O' E|0, 1] HxW) )m= for each I', i.e. { }
( I") Y where I' E TR HxWxD is the feature map of I', i.e., I-=[(T") For each query In of image I, given its corresponding intra-image instance prediction map M,, a total of [ inter-image instance prediction maps {O } r' and the original groundtruth mask Ma(n), we improve the mask prediction loss mask in Eq. 3 as a form of:
((gn,
Σ n=1 (Cintra (Mn,
Mo(n)) (on, + [Z] Engr-inter mask (0%,0))
(5)
where the first term Cintro_mast is exactly the same as Lmask in Eq. which is renamed for address- ing its nature of inspiring intra-scene instance uniqueness. The newly added term Cinter Linter_mask mask is to eliminate the instance ambiguity through inter-scene querying. Thus its training target is an all-zero query-pixe matching matrix 0 of size H W, i.e., each query qu of image I is forced to mismatch the instances in other images I'. Among the several choices for Linter mask: e.g., & loss, & loss, and cross-entropy loss, we find focal loss [66] more favored (cf. $4.3). We speculate this is due the advantage of focal loss in preventing the influence of massive trivial examples, as in our case. To further boost the effectiveness and efficiency of our algorithm, the following strategies are adopted:
External memory: As the capacity of training batch is limited, we follow the recent practice in unsu- pervised representation learning [62, 71-75] to build an external memory to enable large-scale query-instance matching. The memory gathers instance pixel embeddings from several batches. Note that we drop background pixels, as we find their contribution is negligible (sometimes even negative). This also implies that cross-instance discrimination is the core challenge of this task. Sparse sampling: As local image contents are highly correlated, storing all the pixel samples will bring a lot of redundant information, which is useless for discriminative representation learning. With a similar spirit of some dense prediction methods [15, 733, only small set of instance pixels are randomly sampled from each image and fed into the memory. In practice, we find this strategy greatly improves the diversity of the instance samples and eventually benefits the final performance. Instance-balanced balanced sampling: Though effective, the sparse sampling strategy has a side-effect: it hurts the performance on small instances. This is because larger instances have more chance of being sampled for loss computation, enabling our training algorithm to give less priority towards smaller instance. We therefore opt to randomly sample fixed number of pixels from each instance region. As a result, the performance is further improved. All the related experiments can be found in $4.3.
Instance
These designs together form a cross-scene training framework, whose power lies in the query embed- ding uniqueness among plentiful instances. Our framework is elegant and principled: as suggested by Eq. 5, it can complement and integrate with current training paradigm of query-based methods.
Learning with Transformation Equivariance. So far, we have addressed from a fresh perspective of letaset-level instance-unique querying. Next, complete our response from another essential viewpoint: robustness. That is, we expect the segmenters to build robust instance-query correspon- dences that are equívariant against input imagery transformations (e.g.. cropping, flipping, etc).
Our key insight for robustness is derived from the transformation equivariance nature of the task: flip- ping or cropping the image should result in an exact change in the instance segmentation mask. At the first glimpse, this character seems to have been already captured by the widely-used, transformation- based data augmentation technique. Under scrutiny, this view however is erroneous because: the representations (and instance queries) ought to be transformation equivariant first, then it natural to achieve correctly transformed predictions. Yet, the current training strategy puts the cart before
5

---

the horses: with input transformations, the segmenter is only trained to produce correspondingly transformed predictions, but without any reasonable constraints on the representations (and queries).
Given the analysis ahove, we supplement our framework with an equivariance based training objective, encouraging the segmenters precisely and reliably anchor queries instances. Let G be group of transformations (e.g., cropping, flipping, etc). For a query-based instance segmenter, we encourage its feature representation be "equivariant against input imagery transformations" which states:
Vg E G: f(((()) vo g(f()) - 9(
Intuitively, the output representation of f is encouraged to change in the same way to the transfor- mation ! applied to the input I. Similarly, the instance queries are also desired to be transformation equivariant, i.e,, they should be able to properly describe the transformation applied to their counter- part instances. For example, after cropping or flipping an input image, the queries are expected to still robustly capture the changed instance patterns (e.g., shape, location, scale). We formulate this property as:
Vg E G: ( 1 ~ (o(Ma(n)))
where 1 denotes the feature embedding of transformed image (instances) g(I), i.e., 19 (g((): analogously, {g. [ indicates the query embeddings derived from g()). Eq. 7 states that, with a transformed input image g(T). the queries should be properly changed so as correctly match the embeddings of transformed instances 1 This also explains the rationale behind the popular transformation-baase based augmentation technique from a view of equivariant query embedding learning.
Considering both Eq. 6 and 7, we can easily have Vgg G: {(g% ( (M(M)(())) 1== and hence our equivariance based training objective is formulated as (see the bottom part of Fig. 2):
Eequ n. (MMo()))) where W gg
(8)
As such, the desired equivariance property of both feature representations and instance query embed- dings input transformations are formulated in unified training objective. For Eq. 8, the queries ] created for transformed image (I), are first applied for the transformed representation g(T) of T; then the training target is to minimize the difference between the retrieved instance masks {Wge(0, 1| -== and the transformed groundtruth masks (g(M)(m)) 1- Therefore, Lequi can be easily implemented as any existing mask prediction loss, and effortless incorporated into our training framework. For the sake of simplicity, we set Lequi as the same form of Lmask in Eq. 3. In our experiments (cf. $4.3), we will demonstrate that our equivariance constraint can bring significantly larger performance gains than the conventional, transformation-base data augmentation technique.
3.3 Implementation Detail
External Memory and Sampling. For large-scale query-instance matching (cf. Eq. 5), we build an external memory. Due to the limited capacity of our GPU, we afford to maintain queue of 100K pixel samples in the memory. We adopt sparse, mnstance-balancel sampling, i.e., randomly selecting 50 (if possible) pixels from each instance. Note that the samples stored in the memory are only used for the computation of the inter-scene instance discrimination loss Linter_mask in Eq. 5. The external memory is directly discarded after training, causing no extra computation budget during inference.
Training Objective. Ordinarily, our training framework is applicable for existing query-based models, and complementary to their training objectives. In our experiments, we approach our algorithm on four representative query-based methods (34,39-41]bb by adding our inter-scene instance discrimination loss Linter_madd mask (cf. Eq.5) 5) and cquivariance loss Lequi (cf. Eq. 8) into their training targets. As mentioned before, we implement Cinter_mast (cf. Eq. 5) as the focal loss [66], whose nyper-parameters are set to a= 0.1 and y = 2.5. As for Lequi, it is directly set as the same form of the loss used for mask prediction in the base segmenter, i.e., dice loss [67] for [34.39.4] 39.41], and a linear combination of the focal loss and dice loss for [40]. For far comparison, the set of transformations G, which is used for the computation of Lesgui, is [horizontal flipping, random cropping between 0.6 and 1.0]. This con- sistent with the standard data augmentation setup adopted by existing instance segmentation networks. To balance the impacts of our two new training targets, i.e, Linter mask and Lequs we multiply Lequi by coefficient X, which is empirically set 3 (see related experiments in Table 4e & $4.3).

---

4 Experiments
4.1 Experimental Setup
Datasets. We conduct our main experiments on the gold-standard benchmark dataset, i.e., [47, in this field. There are 80 target classes for instance segmentation. As normal, split (115k images) for training and va12017 (5k images) for validation used in our ablation study, The main results are reported on test-dev (20k images). For thorough evaluation, we perform additional experiments on LVISvI [48] on top of SOLOv2 [39]. LVIS is a large vocabulary instance segmentation dataset collected by re-annotating COCO images with 1, 1,203 categories. Thus it is considerably more challenging; it contains total of 100k images of train set under significant long-tain taigld distribution, and relatively balanced val (20k images) and test sets (20k images).
train2017
Base Instance Segmenters, Backbones, and Competitors. To demonstrate the broad applicability and wide benefit of our algorithm, we approach method four widely recognized query- instance segmentation models, i.e., CondInst [34], SOLOv2 [39], SOTR [41], and Mask2Former [40], with diverse backbone networks, i.e., ResNNet-00-101 [45] and Srrin-Small-Basee-LLan
[46]. For
fair comparison, all these models and results are based on our reproduction, following the default hyper-parameter and augmentation recipes in AdelaiDet [76] and MMDetection [77]. In addition to focusing on the comparison with these four base segmentation models, we include group of famous segmenters [6. 11, 12, 15, 33.44] for comprehensive evaluation. In our supplementary material, we report more experiments on two instance segmentation models, i.e., SparseInst [44] and S [43), as well as performance of Mask2Former on panoptic segmentation.
Training. Our implementation based on AdelaiDet [76] and MMDetection [77]. following default training configurations for both COCO and LVISv1 datasets. In particular, all the backbones initialized using corresponding weights pre-trained on ImageNet-IK [78], while remaining layers are randomly initialized. We train models using SGD with initial learning rate of 0.01 for ResNet [45] backboned models and Adam with initial learning rate of le for Swin [46] backboned models. The learning rate is scheduled following the polynomial annealing policy with batch size 16. We use multi-scale training [1 [12, 16, 34, 44]: for ResNet backboned models, the short side of input is randomly chosen within [400, 1200] and long side is set to 1333; for Swin backboned models, we opt random scaling with a factor in [0.1, 2.0] followed by fixed size to 1024 x 1024. ResNet backboned models are trained for 12 epochs, while Swin backboned models are trained for 50 epochs. Note that, compared with the standard, scene-wise training strategy, our algorithm only slows the training speed slightly (~55% see Table 3 & $4.3).
Testing, All our experimental results are reported with 1333 x 800 input resolution. For the sake of fairness, we do not apply any test-time data augmentation. Note that, during model deployment, our training algorithm does not bring any change to network architecture or additional computation cost.
Evaluation Metric. The evaluation metric is the standard segmentation mask AP.
Reproducibility. Our algorithm is implemented in PyTorch. For all our experiments, the training and testing are conducted on eight NVIDIA Tesla A100 GPUs with 80GB memory per-card. To guarantee the reproducibility of our algorithm, our full implementations are made publicly available at https: // gi thub. com JamesI 9/Intt/nceeeeeeeeeee Unique Querying.
4.2 Comparison to State-of-the-Arts
Quantitative Results on COCO test-dev. Table 1 reports comparison results with four base models, i.e., CondInst [34], SOL.Ov2 [39], SOTR [41], and Mask2Former [40], as well as several repre- sentative instance segmentation methods [6, 11, 15, 18,43,44, 44,44,.77 79] on tocot47]test-der test-dev. Without
bells and whistles, our training framework provides remarkable performance improvements over the four query-based based segmenters [34,39-41] with different backbones. In particular, for CondInst [34] and SOLOv2 [39]- - the two famous basey-based approaches that adopt the CNN architecture and dynamic neural network, with ResNet-50 [45] backbone, our algorithm brings 3.1 and 3.2 AP gains, respectively. Similar improvements, i.e., 2.8 and 2.9 in terms of AP, are obtained with ResNet-100 As for SOTR [41] - a very recent method that adopts hybrid structure of Transformer [80] structure and dynamic convolution for query creation, our algorithm also greatly promotes its performance, e.g., 39.6-x4.22 with ResNet-50 and 40.2 40.2-44..6 42.6 with ResNet- 101, in terms of AP. Moreover, when directly applying our algorithm to the newest query- based model - Mask2Former [40], we observe powerful
7

---

Table 1: Quantitative results on COCO [47] test-dev. See $4.2 for details.
Method
Backbone
#Lpoch
AP
APso
APrs
APs
APM
APL
Mask CNNiccv
6
ResNet-101
36.1
57.5
38.6
18.8
39.7
49.5
Cascade Mask R-CNN (wamn)]
ResNet-101
12
37.3
58.2
40.1
19.7
40.6
51.5
HTGevves
ResNet-101
20
39.6
61.0
42.8
21.3
42.9
55.0
Point
RendieveRo [15]
Re
12
36.3
56.9
38.7
19.8
39.4
48.5
QueryInsticevan [18]
ResNet- 101
36
41.0
63.3
44.5
21.7
44.4
60.7
K-Net NeuitSaa [79]
ResNet-101
36
62.8
18.7
42.7
58.8
SOL NeutrSz1 Neutsai)
[43]
Swin-L
50
46.7
72.7
50.6
29.2
50.1
60.9
SparseInstecvwzzl [44]
Re
37.9
59.2
40.2
15.7
56.9
CondInstuccwaa [34]
35.5
55.8
37.7
16.8
39.2
50.6
ResNet--50
+0urs
38.613.1
61.1
41.2
19.712.9
41.1
54.7:4.1
12
CondInst [34]
37.1
58.6
39.3
18.2
40.3
52.9
ResNet-101
Durs
39.9
62.7.4.1
42.4-3.1
20.8
42.3,2.0
55.7-28
SOLOv2 Neut1P200 [39]
35.1
55.5
37.0
13,9
38.4
53.7
ResNet-50
+Durs
38.3
59.614.1
40.6-3.6
17.8
41.8134
56.2
12
SOLOv2 Nouri [39]
36.7
57.5
39.3
16.2
40.2
54.2
ResNet-101
+uurs
39.612.9
60.613.1
43.1-3.8
20.0
43,3
56,9-2.7
SOTR nocvan [41]
39.6
60.7
42.6
10.3
58.7
72.1
ResNet-50
+urrs
42.2
61.9
43.9-2.3
11.0
60.5
73.5
SOTR ucev2 [41]
61.2
43.4
10.2
59.0
73.1
10sNet-101
+ +ura
42.6+2.4
64.1.29
45.8-2.4
11.2
61.2,2.2
75,3
Mask2Formenccveaa) [40]
44.5
68.2
47.7
24.3
49.0
66.6
Swin-S
+0rr
46.7
70.7
50.1-2.4
27.0,2.
50.9,1.9
68.8
Mask2Formencwwzz [40]
44.9
68.5
48.5
24.6
48.7
67.4
Swin-B
+Ours
47.3
71.3
51.0-2.5
28.312.
50.9
69.4-2.0
Mask2Formenccveaa
[40]
50.2
74.8
54.7
29.2
53.8
71.1
Swin-L
100
+urrs
51.8+16
76.0
56.8-2.1
29.9
55.1+13
73.3-22
Table 2: Quantitative results on LVISvI [48] val over SOLOv2 [39].
See $4.2 for details.
Method
Backhone
#Epoch
APs0
AP73
AP,
APa
AP,
Mask R-CNNjocvia [6]
ResNet-50
12
21.7
34.3
23.0
9.6
21.0
27.8
SOLOv2 [39]
21.4
34.0
22.8
9.5
20.9
27.6
ResNet-50
36
Ours
24.1
37.413.4
25.5
13.5
22.8;1.9
29.7:2.1
performance gains across different Swin [46] backbones. For instance, with Swin-B, our algorithm surpasses the vanilla Mask2Former different IoU thresholds, i.e., 71.3 68.5 AP and 51.0 vs 48.5 AP75. Notably, with Swin-L as the Mask2Former's 's backbone, our approach surpasses all the other competitors Table and sets new state-of-the- af-thhe-rt record of 51.8 in AP for instance segmentation on COCO. Furthermore, in spite of different base segmentation network architectures and backbones, our training algorithm consistently improves the performance for object instances of different sizes.
Quantitative Results on LVISv1 [48] val. Table lists the overall performance on LVISv1 val, as well as AP scores the (1~10 instances), common (1 (11~100 instances), and frequen! (> 100 instances) subsets. It can be observed that, our training algorithm pushes a powerful gain by 2.7 on AP, and consistently improves the performance of the base model, SOLOv2, on APr, APc. and AP, by 4.0, 1.9 and 2.1, respectively.
The above experimental results are particularly impressive, considering the fact that the performance gain is purely brought by a new training strategy, without any network architectural modification. Overall, our extensive experiments manifest the effectiveness and wide benefit of our algorithm.
Fig. provides qualitative comparisons of our algorithm against CondInst [34] (left) and Fask2Former [40] (right) on several challenging examples of COCO va va11117. As can be seen, with the help of our training algorithm, CondInst and Formerormer can better distinguish between huddled and similar instances, hence generating higher-quality segmentation maps.
Qualitative Result.

---

Condlnst
Mark
Ours
Ours
Figure 3: Qualitative comparison results on COCO va12017. See $4.2 for details.
Table 3: Analysis of essential components on COCO [47] va12017. See $4.3 for details.
Inter scene Equivariance
Training speed (hour/epnch)
AP
AP50
APTs
APs
APM
APr.
Linter_ mask 5) Lequi (Eq. 8)
35,5
55.9
37.4
16.9
38,9
50.2
1.51
37.6
58.4
40.3
18.2
40.8
54.0
1.57
V
36.5
58.0
39.0
17.1
39.5
51.6
1.52
38.1
60.2
40.6,3.2
18.4
40.9f2.0
54.3
1.59
4.3 Diagnostic Experiment
For thorough examination, we conduct a series of ablative studies on COCO [47] va12017. We adopt CondInst [34] as our base segmentation network with ResNet-50 [45] backbone. To perform extensive experiments, we train models for 12 epochs while keeping other
unchanged.
Key Component Analysis. First, we study the efficacy of the core components of our algorithm, i.e., inter-scene instance discrimination loss Linter_mas (cf. Eq. 5) and equivariance loss Lequi (cf. Eq. 8). In Table 3, 1st row gives the performance of our base segmenter CondInst. For 2nd and 3rd rows, the scores are obtained by adopting Linter mask and Lequi individually. And 4th row reports the scores of full algorithm. i) 1st vs 2nd row: Linter mask leads to notable performance improvements against the baseline (e.g., -37.6 AP). This verifies our first core hypothesis, i.e., learning large-scale instance separation benefits discriminative query embedding learning. ii) 1st vs 3rd row: Lequi greatly boots the performance of the baseline (e.g..
AP), evidencing our second core hy hypthhesis, i.e., exploiting the transformation equivariance nature of the task facilitates robust query-instance matching. iii) 1st vs 2nd vs 3rd vs 4th row: combing all contributions together results in the largest gain over the baseline (e.g., 35.5 +381 AP). This suggests that Cinter_mast and Lequi are able to work in collaborative manner, and confirms the effectiveness of our overall algorithmic design.
Training Speed. We also report training speed in Table 3; related statistics gathered on eight A 100 GPUs with a batch size of 16. We can find that, compared with the current scene-wise training regime, our cross-scene training algorithm only brings slight delay (~5%). More specifically, the computation of our inter-scene instance discrimination loss Linter mask takes relatively more time, due to the use of an external memory for acaee-ccale query-instance matching.
Next we study some core designs for inter-scene instance discrimination loss Linter mask (cf. Eq. 5). The results in Tables 4a, 4b, and 4c reported without considering equivariance loss Lequi (gf. 8).
Loss Design for Inter-Scene Instance Uniqueness Learning. In Table 4a, we investigate four different implementation forms of our inter-scene instance discrimination loss Linter masks namely & loss, f2 loss, cross-entropy loss, and focal loss [66]. We can find that focal loss is more favored. In our case, most instance (pixel) samples from other scenes can be easily recognized as negative, causing a huge imbalance between hard and easy negative samples. Therefore, training under such an environment results in & loss, l2 loss, and cross-entropy loss focusing more easy samples. In contrast, focal loss drives the training more towards the sparse set of hard negative samples, thus preventing the gradient of Linter_mask from being dominated by the massive, easy negative samples.
Sampling Strategy. We further study the impact of the three sampling strategies (cf. $3.2) for the computation of Cinter_mask, namely dense sampling (storing all the pixels of each image into the memory), sparse sampling (randomly sampling a small set, egg., 0.5k or 1.0k, of pixels from each image). and instance-balanced sampling (randomly sampling small fixed-size set, e.g., 10 or 50, of pixels from each instance). Table 4b proves that, sparse sampling works overall better than dense sam- pling, as it improves the diversity of the stored samples. However, sparse sampling leads to inferior

---

Table 4: A set of ablative studies on COCO [47] va12017. The adopted algorithm designs and Hyper-parameter settings are marked in red. See $4.3 for details.
Linter mark
APs APM A
sampling
AP
APs APM APr
capacity
AP
[APs
M
APL
35.5
16,9 38,9 50.2
dense
37.1
18.0 40.4 53.0
16.9
38.9
50.2
6 loss
36.0
17.1 39.2 51.7
sparse paxeVing.)
37.2
17.8
40.5 53.2
mini-atch (whs mennty)
36.9
17.8
40.1
53.1
62 loss
35.9
17.6
39.1 51.6
sparse (1 ik pixeliig
37.3
17.9
40.5
53.4
10k pixels
37.0
17.9
40.3
53.2
cross-entropy
37.3
18.1 40.6 53.3
instance-bale ballnced
(10 pixclliss.)
37.4
18.7
40.5
53.6
50k pixels
37.3
18.1
40.4
53.6
local loss
37.6
18.6 40.6 54.0
instance balanced
70k pixels
37.5
18.5
40.4
53.7
pisellas.)
37.6
18.6
40.6 54.0
100k pixels
37.6
18.6
40.6 54.0
(a) inter-scene instance discrimi- nation loss Linter mask (cf. 5)
(b) sampling strategy
(c) memory capacity
transformation equivariance
AP
APS
APM
APL
coefficient
AP
APs
APM
API.
w/o Aug. w/ Aug. Wig. Ju ())
35.3
16.2
39.2
49.9
1
37.0
17.9
40.4
52.1
35.5
16.9
38.9
50.2
3
37.6
18.6
40.6
54.0
feature only (T& g(T))
36.3
16.5
39.5
51.4
5
37.5
18.3
40.7
53.6
Cegus (1g (())), (M(M))))))) (.))
36.5
17.1
39.5
51.6
10
37.3
18.1
40.8
53.3
(d) transformation equivariance modeling
(e) coefficient Conter and Legui
performance on small instances, e.g., 18.0 vs 17.8 APs for 'sparse pixeVing)), as smaller instances balanced sampling inherits sparse
are less likely sampled for loss computation. Fortunately, sampling's 's desired merits but without its defects, hence yielding more impressive results.
Memory Capacity. Table 4c shows the influence of the capacity of the external memory. 1st row gives the results of the base segmenter. 2nd row lists the scores obtained by computing Linter_ment mmak within each mini-batch, which are already much better, e.g., 35.5 vs 36.9 AP. 3rd 5th rows demonstrate that, i) with the aid of an external memory, the performance can be further boosted; and ii) enlarging memory is always helpful. These results verify our key idea of learning large-scale query instannn matching. Note that the optimal configuration, i.e., 100k pixels, does not yet reach the point of performance saturation, but rather the upper limit of our hardware's computational budget.
Transformation Equivariant Learning. We next study our equivariance loss design Lequi (cf.Eq. Eq. The results in Table 4d are reported without considering the inter-scene instance discrimination loss Linter_man mask (cf. Eq. 5). The first two rows respectively present the performance of the base segmenter w/o and w/ transformation-oe based data augmentation. As we mentioned before, training with current data augmentation technique can be viewed as learning equivariant query embeddings (cf. Eq. 7). In 3rd row we show the performance by only encouraging equivariant image representation learning (cf. Eq.6). Comparing these three baselines with our final equivariance loss Legui that simultaneously addresses equivariant image representation and query embedding learning, we can find: i) exploring the equivariance nature of the task indeed essential; and ii) our equivariance loss Legui is much more effective than current transformation-hant
data augmentation technique (e.g., 36.5 vs 35.5 AP).
Loss Term Coefficient. For completeness, the results with different coefficients, which control the balance between our two training objectives, i.e., Linter Linter_mask mask and Lequi, are reported in Table 4e.
5 Conclusion and Discussion
Aiming at sharpening the instance discrimination ability of query-based segmenters, we devise a novel framework that formulates two crucial properties of the query- query-instance relationship, i.e., dataset- level uniqueness and transformation equivariance, as network training targets. This is achieved by compelling segmenters to i) build exclusive instance-query matching throughout the entire training dataset, and ii) learn equivariant instance-query matching with respect to geometric transformations. Extensive empirical analysis demonstrates that our method is flexible yet powerful, allowing it to benefit the existing and growing body of qucry-based instance segmentation methods. We argue deductively that the proposed training framework has the potential to be applied to broader range of dense prediction tasks, i.e., query- based detection and panoptic segmentation. These questions remain open for our future endeavor.
Acknowledgement. This work was partially supported by ARC DECRA DE220101390.
10

---

References
[1] Pedro O Pinheiro, Ronan Collobert, and Piotr Dollar. Learning to segment object candidates. In NeurlPS, 2015. 1,2
[2] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumas for object segmen- tation and fine-grained localization. In CVPR, 2015. 1,2
[3] Jifeng Dai, Kaiming Hc, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR, 2016. 1.2
[4] Pedro O Pinheiro. Tsung-Yl Yi Lin, Ronan Collobert, and Piotr Dollár. Learning to refine object segments. In ECCV, 2016. 1
[5] Bernardino Romera- Paredes and Philip Hilaire Sean Torr. Recurrent instance segmentation. ECCV,
2016. 1,2
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 1, 8 [7] Mengye Ren and Richard $ Zemel. End-to-end instance segmentation with recurrent attention. In CVPR, 2017. 1,2
[8] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmenta- tion. In CVPR, 2018. 1,2
[9] Liang- Chen, Alexander Hermans, George Papandreou, Florian Schroff. Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features.
In CVPR, 2018. 1,2
[10] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cmn. In CVPR, 2019.
[11] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: high quality object detection and instance segmenta- tion. IEEE TPAMI, 4385):1483-1198, 2019. 2, 7,
[12] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, 2019. 1,2, 7,
[13] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan. Blendmask: Top-down meets bottom-up up for instance segmentation. In CVPR, 2020. 1,2,4 4
[14] Tianheng Cheng. Xinggang Wang. Lichao Huang, and Wenyu Liu. Boundary-preserving mask r-cnn. In ECCV, 2020. 1.2
[15] Alexander Kirillov, Yuxin Wu, Kaiming Hc, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, 2020. 1,
[16] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion-aware instance segmentation with overlapping bilayers. In CVPR, 2021. 2,
[17] Thang Vu, Hacyong Kang, and Chang D Yoo. Scnet: Training inference sample consistency for instance segmentation. In AAAI, 2021. 1,2
[18] Yuxin Fang, Shusheng Yang. Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries. In ICCV, 2021. 1, 2, 4, 8,8
[19] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.
[20] Anurag Amab and Philip HS Torr. Bottom-up instance segmentation using deep higher-order erfs. arXiv preprint ar Xiv: 02583, 2016. 1,2
[21] Anurag Arnab and Philip HS Tomr Pixelwise instance segmentation with dynamically instantiated network. In CVPR, 2017. 1,2
[22] Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. In CVPR, 2017. 1,
[23] Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. Sgn: Sequential grouping networks for instance segmentation. In ICCV, 2017. 1,2
[24] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In NewrIPS, 2017. 1,2
[25] Shu Kong and Charless C Fowlkes. Recurrent pixel embedding for instance grouping. In CVPR, 2018. 1.2
[26] Davy Neven, Bert De Brabandere, Mare Proesmans, and Luc Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. CVPR, 2019. 1,2
[27] Naiyu Gao, Yanhu Shan, Yupei Wang Xin Zhao, Yinan Yu, Ming Yang, and Kaigi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In ICCV, 2019. 1,2
11

---

[28] Tianfei Zhou, Wenguan Wang. Si Liu, Yi Yang, and Luc Van Gool. Differentiable multi-granulann human representation learning for instance-aware human semantic parsing. In CVPR, 2021.
[29] Jifeng Dai, Kaiming He, Yi Li, Shaoqing Ren, and Jian Sun. Instance-sensitive fully convolutional
networks. In ECCV, 2016. 1,2
[30] Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware semantic segmentation. In CVPR, 2017. 1,2
[31] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár. Tensormask: A foundation for dense object
segmentation. In ICCV, 2019. 1,2
[32] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In ICCV, 2019. 2,
[33] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV, 2020. 1,2
[34] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, 2020. 1, 2, 4, 6, 7, 8, 9,
[35] Enze Xie. Peize Sun, Xiaoge Song. Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In CVPR, 2020. 1,2
[36] Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor- free instance segmentation. In CVPR, 2020. 1.2
[37] Yuqing Wang, Zhaoliang Xu, Hao Shen, Baoshan Cheng, and Lirong Yang. Centermask: single shot instance segmentation with point representation. In CVPR, 2020. 1,
[38] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations.
In ECCV, 2020. 1,2
[39] Xinlong Wang, Rufeng Zhang. Tao Kong. Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. In NeurlPS, 2020. 2, 4, 6, 8, 14
[40] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 1, 2, 3, 4, 6, 7, 8, 14
[41] Ruohao Guo, Dantong Niu, Liao Qu, and Zhenbo Li. Sotr: Segmenting objects with transformers. In ICCV, 2021. 1, 4, 7, 14
[42] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurlPS, 2021. 1, 2,4,
[43] Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects by learning queries. In NeurlPS, 2021. 1, 2, 3, 4, 7,8
[44] Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Wenqiang Zhang, Qian Zhang, Chang Huang. Zhaoxiang Zhang, and Wenyu Liu. Sparse instance activation for real-time instance segmentation. In CVPR, 2022. 1, 2, 7,
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 7,9
[46] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 2, 7,8
[47] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 2, 7, 8, 9,
[48] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation.
In CVPR, 2019. 2, 7,8
[49] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017. 2
[50] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection.
In ICCV, 2019. 2
[51] Yiming Cui, Liqi Yan, Zhiwen Cao, and Dongfang Liu. Ti-blender: Temporal feature blender for video object detection. In ICCV, 2021. 2
[52] Wenchao Gu, Shuang Bai, and I. Lingxing Kong. A review on 2d instance segmentation based on deep neural networks. Image and Vision Computing, 2022. 2
[53] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. In NeurIPS, 2016. 2
[54] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3.4
12

---

[55] Guo-Jun Qi, Liheng Zhang, Feng Lin, and Xiao Wang. Learning generalized transformation equivariant representations via autoencoding transformations. IEEE TPAMI, 2020.
[56] Geoftrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks, 2011. 3
[57] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In ICRL, 2018.
[58] Taco Cohen and Max Welling. Group equivariant convolutional networks. In ICML, 2016.
[59]
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In CVPR, 2017.
[60] Uwe Schmidt and Stefan Roth. Learning rotation-aware features: From invariant priors equivariant descriptors. In CVPR, 2012.
[61] Kihyuk Sohn and Honglak Lee. Learning invariant representattin with local transformations. In ICMI., 2012. 3
[62] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 3,5
[63] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 3
[64] Xinlci Chen and Kaiming Hc. Exploring simple siamese representation learning. In CVPR, 2021. 3
[65]
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačíć. Equivariant contrastive learning. In ICLR, 2022. 3
[66] Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017. 4, 5,
[67]
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016. 4,
[68] Dongfang Liu, Yiming Cui, Liqi Yan, Christos Mousas, Baijian Yang, and Yingjie Chen. Densernet:
Weakly supervised visual localization using multi-scale feature aggregation. In AAAI, 2021. 4
[69] Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Rank & sort loss for object detection and instance segmentation. In ICCV, 2021. 4
[70] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017. 4
[71] Xun Wang, Haozhi Zhang. Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. CVPR, 2020.
[72] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0 04299,,00. 2020.
[73] Wenguan Wang. Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool, Exploring tross-image pixel contrast for semantic segmentation. In ICCV, 2021. 5
[74] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest. centroids. arXiv preprint arXiv:2209.07380 2022.
[75] Zhiyuan Cheng. James Liang. Hongjun Choi, Guanhong Tao, Zhiwen Cao, Dongfang Liu, and Xiangyu Zhang. Physical attack on monocular depth estimation with optimal adversarial patches. In ECCV, 2022. 5
[76] Zhi Tian, Hao Chen, Xinlong Wang, Yuliang Liu, and Chunhua Shen. AdelaiDet: A toolhox for instance- level recognition tasks. https //t. io/ wtt.:oo//eeaae adelaidet, 2019. 7
177] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDelection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: /0t..//90007155... 2019. 7
[78] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012. 7
[79] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NewrlPS, 2021. 7,8
[80] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017. 7
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] Our main claims accurately reflect the paper's contributions and scope.
(b) Did you describe the limitations of your work? [Yes] The limitations are discussed in $D of the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The potential negative societal impacts discussed in $D of the supplemental material. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have already read the ethics review guidelines and ensured that our paper conforms them.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results[[N/A] Our work does not involve any theoretical result.
(b) Did you include complete proofs of all theoretical results?[N/AA] Our work does not involve any theoretical result.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] The full implementations are made open source at https:/ / ////gtttbbb.. Instance Unique_Querying The pseudo-code is also included in the supplemental material. The datasets used in this work are online publicly available and well cited in the paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We report the training details and hyperparameters in $4.1.
(c) Did you report error bars (c.g., with respect the random seed after running periments multiple times)? [No] We follow the standard evaluation protocol this report and compare the results. Running all our experiments multiple times is too expensive to afford.
[34,39444444
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We report our hardware specification in the Reproducibility part of $3.3.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all used codes, data, and models in this paper.
(b) Did you mention the license of the assets? [Yes] CondInst and SOLOv2 implemented based on their released code (https:/ htt///gittttbb wwm/aimmmmm/...ee.c.. fofa AdelaiDet) with Apache2.0 license. SOTR is implemented based its released code (https://ge) /tt::://ttttttt com/easton. catn-cca/SSOn /SOTR) with MIT license. Mask2former is implemented based on the reproducing by MMDetection /tttps:/giithb com/ open/ope--oo mmlab/ mndetect ietecttiionnnteeeeeeeee master conf ga maskkffrkkrrrrrr with Apache2.0 license.
(c) Did you include any new assets either in the supplemental material [Yes] SparseInst implemented based on its released code (https: //pp:::/gii thub. com/ hustvl/SparseInst) with MIT license. SOLQ implemented based on its released code (https: /tctts::/githut
URL?
www.riieeereeeseeeeeeee /SOLQ) with Apache2.0 license.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] MS COCO dataset public and licensed under Creative Commons Attribution 4.0 License.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] The data used in our paper neither contain any personally identifiable information nor offensive content.
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "z_EgHqToxBK", "reviewer": "Reviewer_jN1s", "summary": "This paper proposes a new training framework that exploits query embedding learning. Dataset-level uniqueness and transformation equivariance are introduced and demonstrate promising results on the benchmark datasets.", "questions": "In Ln 158, uniqueness and robustness are named as two crucial properties. Can these be consistent with the statement in abstract Ln 5?\n\nIf the equivariance is estimated on the feature representation, what makes it different to traditional concepts, e.g. invariance?", "limitations": "Despite the novelty of introducing equivariance, the in-depth analysis of the mathematical properties is limited. Only empirical results cannot fully reflect the rationale behind the equivariance eqution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Framework contribution: this work brings a new paradigm shift that goes beyond inner-scene training to an inter-scene level query embedding separation.\n\n+ Introducing equivariance is intuitive and the results are convincing.\n\n- The evaluation is somewhat weak. Only COCO is adopted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pEIfrRu8jOA", "reviewer": "Reviewer_Zz8f", "summary": "This work proposes a novel instance segmentation training paradigm that can be combined with any model. Two properties are explored: dataset-level uniqueness and transformation equivariance. This training is combined with quite a few SOTA models, and it is shown that a significant gain of 2-3 AP on COCO. Training is only 5% slower than standard training, with no speed loss at instance. ", "questions": "I asked my questions in weaknesses. Authors should focus on those in the rebuttal.", "limitations": "Nothing to add here.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This training paradigm is novel as far as I can tell, it is properly motivated, clearly explained and both additions are orthogonal, which results in experiment section clearly show\n+ When I started reading the paper, I expected that training cost will be much higher than baseline, but upon reading the implementation strategies (namely external memory, sparse sampling and instance-balanced sampling) I was convinced that the loss in speed will be minor, and it was, it is just 5%! Kudos for the engineering aspect of the paper\n+ Experimental section is very well designed. Authors compare with a lot of SOTA approaches, they incorporate their method with 4 different representative models. Then, they clearly explain the effect of inter-scene and equivariance, they are both providing performance boost and are orthogonal, so combined they achieve biggest jump. Finally, ablation over hyperparams is very good, it shows that the newly added hyper-parameters are fairly robust (at least on COCO dataset), and it helps the reader select them on their problem.\n+ Performance gains are very good, hard to be ignored. Performance of the best result is SOTA. \n\nWeaknesses:\n- Whole paper evaluated on only one dataset. I do know COCO is standard, but for a paper that claims to improve training paradigm, I think maybe one more dataset should be used. Not to achieve SOTA necessarily, but to show benefits on top of a good model. Eg, evaluation on LVIS would greatly improve the paper, in my opinion.\n- Minor: When reading Mask2Former paper, it seems their equivalent COCO result is 50.1, not 48.5 like here. Can authors explain in more detail what is the difference in their implementation and why this happens?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dbmNjpoutGV", "reviewer": "Reviewer_XmTr", "summary": "This paper proposes using two ways to improve the performance of query-based instance segmentation network. First, by using the queries to retrieve the corresponding instance from the whole training dataset, the segmenters are forced to learn instance-uniques queries. Second, by performing geometric transformations and encouraging the network predictions to be equivariant, the authors expect to learn more robust instance-query matching. Experiments show that both ways can improve the instance segmentation effectively. ", "questions": "See the weakness part.", "limitations": "The authors didn't discuss the limitations and potential negative societal impact of their work. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n-The idea is simple and reasonable.\n\n-Overall, the paper is easy to read and understand.\n\n-Experiment results are good. \n\nCons:\n-The relationship between the proposed two contributions seems not to be clear. And the equivariance-based augmentation seems not to be tailored for instance segmentation, and even not to be tailored for query-based segmentation framework. I think it is a common augmentation trick which may not be suitable to claim it as a contribution of this paper. \n\n-What is the detailed format of cross-entropy loss and focal loss for $\\mathcal{L}_{inter\\_mask}$？", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F_zZIFHbDX2", "reviewer": "Reviewer_2dJn", "summary": "- The paper proposes to enhance the dataset-level uniqueness and transformation equivariance of queries in the query-based segmentation methods.\n- The proposed method essentially adopts some technics in contrastive learning to learn discriminative queries.\n- The method consistently improves baseline methods like CondInst, SOLOv2, SOTA, and Mask2Former.", "questions": "1. Why color jittering and blur is not suitable? It is quite common in contrastive learning [58,59,60] to use both geometric and photometric transformations, therefore, intuitively the reviewer does not understand why it is not suitable.\n2. Methods like MaskFormer, Mask2Former, and K-Net are not only used for instance segmentation but also panoptic segmentation methods, does the proposed method can have a more generic form that works for panoptic segmentation or it can only be used for instance discrimination?", "limitations": "1. As asked in the questions, if the methods only applies to instance segmentation, it might be better to claim its scope.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. The paper exhibits consistent significant improvements over many methods including CondInst, SOLOv2, SOTR, and Mask2Former.\n2. The paper reveals a new direction of improvements over query-based instance segmentation methods by enhance the discrimination ability of queries.\n\n## Weaknesses:\n1. There are too many equations that makes the paper hard to follow. For example, the notion of $O^I_{n}$ in eq.5 and  $\\hat{W^{g}_{n}}$ in eq. 8\n2. May also show the iteration time and GPU memory cost in Table 3 (b) and (c) to show the trade-off.\n3. K-Net [a] is not mentioned and compared in the paper, which has better speed-accuracy trade-off than SOLOv2 and CondInst.\n4. The description is not accurate and should be updated. This paper essentially improves methods that use dynamic kernels, i.e., ‘kernel-based’ methods rather than 'query-based' methods. SOLOv2 [37], CondInst[32], SOTR[39], K-Net [a], MaskFormer[40], Mask2Former [38] they essentially all use some strategy to predict content-aware kernels then use the kernels to perform convolution with the feature map to produce masks. This is explicitly described in the original paper of SOLOv2, CondInst, SOTR, and K-Net, although MaskFormer and Mask2Former describe them as ‘mask embeddings’.\n5. The augmentation techniques are not sufficiently studied, i.e., only geometric transformation is studied in Table S2 while photometric transformations and the detailed parameters like cropping range are not studied.\n[a] K-Net:Towards Unified Image Segmentation, NeurIPS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept