SOLQ: Segmenting Objects by Learning Queries
Authors: Bin Dong
Fangao Zeng
Tiancai Wang
Xiangyu Zhang
Yichen Wei
Venue: NeurIPS 2021 Poster (2021)
Keywords: instance segmentation
Transformer
compression coding

Abstract: In this paper, we propose an end-to-end framework for instance segmentation. Based on the recently introduced DETR, our method, termed SOLQ, segments objects by learning unified queries. In SOLQ, each query represents one object and has multiple representations: class, location and mask. The object queries learned perform classification, box regression and mask encoding simultaneously in an unified vector form. During training phase, the mask vectors encoded are supervised by the compression coding of raw spatial masks. In inference time,
mask vectors produced can be directly transformed to spatial masks by the inverse process of compression coding. Experimental results show that SOLQ can achieve state-of-the-art performance, surpassing most of existing approaches. Moreover, the joint learning of unified query representation can greatly improve the detection performance of DETR. We hope our SOLQ can serve as a strong baseline for the Transformer-based instance segmentation.

SOLQ: Segmenting Objects by Learning Queries
Bin Dong
Fangao Zeng"
Tiancai Wang MEGVII Technology
Xiangyu Zhang
Yichen Wei
(dongbin, wangti inncai,
Abstract
In this paper, we propose an end-to-end framework for instance segmentation. Based on the recently introduced DETR [1], our method, termed SOLQ, segments objects by learning unified queries. In SOLQ, each query represents one object and has multiple representations: class, location and mask. The object queries learned perform classification, box regression and mask encoding simultaneously in an unified vector form. During training phase, the mask vectors encoded are supervised by the compression coding of raw spatial masks. In inference time, mask vectors produced can be directly transformed to spatial masks by the inverse process of compression coding. Experimental results show that SOLQ can achieve state-of-the-ee performance, surpassing most of existing approaches. Moreover, the joint learning of unified query representation
greatly improve the detection performance DETR. We hope our SOLQ can serve as a strong baseline for the Transformer-basee instance segmentation. Code available https: //g thub. con/megvvii http://vvvirrseerrcc//ooecc rwweercheeSS....
Introduction
Instance segmentation, serving one of visual detection tasks, not only locates instances of different categories but also generates pixel-level mask for each instance. State-of-the-ar instance segmentation methods [2, 3, 4, 5] follow the two-stage paradigm, which first performs object detection and then segments the masks within detected boxes by RolAlign [2]. Those methods are relatively easy to be optimized thanks to the deployment of mature object detectors [6, 7]. However, the segmentation branch heavily relies on the detection branch, making hard to achieve better joint learning of multiple tasks. Some recent works [8, 9, 10, 11] build instance segmentation frameworks
top of anchor- free object detectors [12, 13] to remove the ROI-Cropping operation, reducing the effect of feature misalignment. For example, CondInst [9] based on FCOS [12] employs dynamic convolutions [14] to perform instance segmentation. YOLACT [8] models the instance segmentation as combination of prototypes weighted by learned mask coefficients for each anchor. However, the weights of dynamic convolutions or the mask coefficients are still generated by instance proposals.
Regardless of the bounding boxes, SOLO [15] introduces the notion of "instance categories" and segments objects by locations. SOLOv2 [16] further solves the problem of incfficient mask represen- tation by introducing dynamic convolution and so-called matrix Mon-Maximum Suppression (NMS). Though SOLO directly outputs instance masks based on locations, hand-crafted post-processes, like NMS, are still required remove duplicated predictions. Also, the performance of small objects is far from satisfactory due to the imbalance of location samples between the large and small objects. Building an end-to-end instance segmentation framework is still remaining problem.
Our work inspired by DETR [1], which first proposes the end-to-end solution for object detection. In DETR, object detection is regarded as set prediction problem and objects represented by
"Equal contribution. This work is supported by The National Key Research and Development Program of China N...2017YFA00000000 and Beijing Academy of Artificial Intelligence (BAAI).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

learnable query embeddings. How to encode the spatial binary mask into such an end-to-end system an opening question. As shown in Fig. (a)), DETR further extended to panoptic segmentation by directly reshaping the learnable embeddings into spatial domain and building FPN-style [17] network to produce the final mask predictions. However, both the Transformer encoder and decoder fail to model the spatial information well. Therefore, it is inappropriate to generate the spatial mask based on such query embeddings. Besides, the spatial mask labels used for supervision are of large resolutions, which results in high computation cost and makes it separated to learn the detection and mask branches". So we need to find one mask representation that satisfies the following conditions:
1) The representation can naturally convert object mask from spatial domain to embedding domain; 2) The process that encodes the spatial masks into embeddings should be reversible; 3) Mask embeddings encoded can keep the principle components of spatial mask. We turn to methods in literature for help and surprisingly find that classical compression coding methods (e.g. Sparse Coding [18]) just satisfy these conditions mentioned above. The mask embeddings can be simply generated from the learnable queries. In training phase, ground-truth spatial mask of each instance can be projected into low-dimensional mask embedding by compression coding and the mask embeddings are used to supervise the learning of predicted mask embeddings. In inference phase, binary spatial mask can be reconstructed from predicted mask embedding by the inverse process of compression coding.
With these analysis, we explore how to better encode the spatial mask into the end-to-end object detectors in this paper. Based DETR. our proposed method, termed SOLQ, segments objects by learning queries. In SOLQ, we formulate the instance segmentation as the joint learning of unified query representation (UQR). The UQR learned can be used to perform parallel predictions for three sub-tasks (classification, localization and segmentation) simultaneously and all predictions obtained in a regression manner (see Fig. ((b).. In this way, SOLQ directly outputs instance masks together with corresponding class confidences and box coordinates. The learning of UQR can be divided into two parts: generating instance-aware query embeddings and joint supervision of multi-task learning. Specifically, all candidate instances are initialized with several learnable queries, which interact with extracted image features in Transformer decoder to produce the instance-aware query embeddings. The instance-aware query embeddings are further input to three branches of sub-tasks, which contains several linear projection layers, to generate three sub-task vectors. For the classification and regression branches, we follow the same supervisions as in DETR [1]. For the mask branch, we conduct implicit supervision with the help of mask compression coding mentioned above.
To summarize, our contributions are:
We propose an end-to-end framework for instance segmentation based on DETR. SOLQ formulates the instance segmentation as the joint learning of UQR. In UQR, the mask representation can be converted from spatial domain into embedding domain, which is consistent with the learnable query embeddings in DETR.
Experiments show that SOL with ResNet101 achieves 40.9% mask AP and 48.7% box AP on the challenging MS COCO dataset [19] without bells and whistles, outperforming SOLOv2 by 1.2% mask AP and 6. box AP. It worthy noting that SOLQ can improve 2.0% in box AP compared to DETR thanks to the joint learning of UQR
2
Related Work
Instance Segmentation Instance segmentation is a classic but challenging computer vision task. required to output each object instance in image with instance-level category label and localization, along with pixel-level mask simultancously. Currently, there are mainly three categories of instance segmentation methods: top-down, bottom-up and directly-predict methods. Top-down approaches [20, 2, 8, 21, 21,22, 23, 3, 5, 24] follow the detecr-then-segment pipeline. They first generate bounding boxes by object detectors and segment the masks by ROIAlign [2] or dynamic convolutions [9]. Bottom-up methods [25, 26, 27, 28] learn per-pixel embeddings via semantic label and then cluster them into instance groups For pirectty-predict methods. PolarMask [29] employs polar coordinates to represent mask contours. Latest SOLO [15] and SOLOv2 [16] directly segment the objects by locations without dependence on bounding boxes or embedding learning. Query Inst [24] and ISTR [30] extend the Sparse RCNN [31] to perform end-to-end instance segmentation. In this paper, we
2 Trrin the object detector first and then freeze the weights of object detector train segmentation branch.

---

explore an end-to-end instance segmentation solution by learning an unified query representation without any post-processing procedures, like Non-Maximal Suppression (NMS).
Transformer in Vision Transformer [32] introduces the self-attention mechanism to model long- range dependencies, and has been widely applied in natural language processing (NLP). Recently, several works attempted to involve the Transformer architecture into various computer vision tasks and showed promising performances. The non-local block [33] is first proposed enhance video recognition by aggregating spatial information. After that, CCNet [34] further extends the self- attention via sparse attention in semantic segmentation. DETR [1] and Deformable DETR [35] adopt learnable queries and Transformer architecture together with bipartite matching to perform object detection in end-to-end fashion, without any hand-crafted process such as NMS. IPT [36] proposes a transformer based pretrained network for low-level image processing. ViT series [37, 38, 39, 40] take an image as sequence of patches and achieve the cross-patch interactions by Transformer architecture in image classification.
Compression Coding in Vision Consider the advantage of low-dimension representation and less computation cost, some recent works have attempted to introduce compression coding methods, like Sparse Coding [41], Principal Component Analysis (PCA) [42] and Discrete Cosine Transform (DCT) [43], into computer vision field. For image classification, [44] takes the DCT coefficients obtained from RGB images as the inputs of convolutional neural networks (CNNs) reduce the communication bandwidth between CPU and GPU. DCT-Mask [45], based on Mask R- R-CNN, employs DCT supervision to produce high-quaie qualltyiitt mask representation. Analogously, [46] performed semantic segmentation on the DCT representation and fed the rearranged DCT coefficients to CNNs. MEInst.. [21] and ISTR [30] encode binary masks into fixed-dimensional mask vectors produced by PCA.
3 Method
3.1 Reviewing DETR
Recently, DETR [1] succeed in object detection. formulates object detection as a set prediction problem and introduces object queries, set of learnable embeddings, to represent objects. In DETR, each object query predicts an object for a given input image and set prediction loss is adopted to achieve one-to-one matching between the predicted and ground-truth objects in training phase. Further, the transformer encoder-decoder architecture is employed to model the relation between query embeddings and instances for better one-to-one set prediction.
Set Prediction Loss Let = (c, b) and (é, b) denote the ground truths y and the set of predictions U. respectively. c, è E RJ are the corresponding class labels and predicted class scores, where and S are the object number and class number. b, E Rxx4 are the corresponding ground-truth and predicted box coordinates. E S2 is the assignment between the ground truths and predictions. Then the optimal one-to-one assignment w' can be calculated by bipartite matching [47] as:
w- argmin (y. w(ÿ))
wED,
where the bipartite matching loss for object detection Last can be summarised as:
Lact (y, 1) Nots (c, + Ar, Li, (b, + Agion Lgioou (b, 6)
Here Lats denotes the focal loss [48] for classifications, Lr., and Ligin are L1 loss and generalized IoU loss [49] for box coordinates, respectively. Aets. Ar., and İigiou are corresponding coefficients.
Extension on Segmentation As shown in Fig. 1(a), DETR is further generalized to panoptic seg- mentation task by adding multi-head attention (MHA) and FPN-style CNN after the Transformer decoder. Features from the encoder and learned query embeddings from the decoder are reshaped to spatial domain and then interact in MHA. The produced features are then gradually upsampled to the image size by the FPN-Style CNN to obtain spatial masks. It's easy to adapt this framework to perform instance segmentation by cropping instance masks within detected bounding boxes.
Object Representation DETR, objects are represented as set of object queries. Object queries are initialized by the learnable query embeddings and then interact with the image features in the transformer decoder to update their representation. Finally, object query, as an unified representation, is directly used to classify and localize objects. However, the representation of spatial mask is built by
3

---

compulsorily reshaping the query embeddings to spatial domain in instance segmentation task. Such design leads different representation forms compared detection branch. Besides, the two-stage training process makes DETR fail to enjoy the benefit from multi-task learning.
3.2 Network Architecture
To encode the spatial mask into the end-to-end object detector in an unified form, we present simple but efficient framework for instance segmentation based on DETR. The unified query representation (UQR) is proposed to perform instance-level localization and pixel-level segmentation simultaneously. The compression coding is further introduced to project the spatial mask into embedding domain for high-quality and efficient mask representation. The overall architecture of SOLQ is showed in Fig. 1(b). SOLQ can be divided into three parts: feature extraction network, Transformer decoder and unified query representation. We will describe our method in detail as follows.
Cls & Box
Cls Box & Mask
Image
Res2
Res3
Res4
Res5
or()
MI
Image
Rea2
Res3
Resd
ResS
FF()
x6
Drop
Frozen
Mask
x2
DConv
x2
DCanv
62
DConv
x2
DCony
MIIA
Identity & Add
7()) Encoder
Initial Learnable Queries q²
Reshape
X(()) Transformer Decoder
FPN-style instance segmentation
(b) SOLQ directly-predict style segmentation
Figure 1: Architecture comparison between the naive DETR and the proposed SOLQ for instance segmentation. "Res2-Res5" are different stages of ResNet [50]. "DConv" and "MHA" are deconvolu- tion [51] layer and muultiheead attention, respectively. "MLPs" means multi-layer perceptions and "UQR" denotes unified query representation, which introduced in detail via Fig. 2. qd is initial learnable object queries. "Frozen" means that train object detector firstly, and then freeze weights of object detector to train instance segmentation branch, separately.
Feature Extraction Network The feature extraction network consists of the backbone and Trans- former encoder. Given an image I E R XXWXXX x3, ResNet [50] is used as the backbone to extract basic feature map _² E CCx $2 x $2, where H, W, C are the height, width and channels of feature map, respectively. Then the basis feature x is fed into K Transformer encoder layers get the refined HW 32"h
feature map xx E IR
via fa* = F (m*- }) iteratively. Each Transformer encoder layer Fk (-) is composed of a multi-head self-attention (MHSA) and a feed-forward network (FFN).
C'x
Transformer Decoder Given the learnable object queries, generate the instance-aware query embeddings for the unified query representation by the Transformer decoder. In details, a set of learnable object queries 9 E R are firstly randomly initialized. Then the initial object queries d interact with the refined feature map K in K Transformer decoder layers to obtain instance- aware query embeddings gx E R by {q" H. (g ,x*)) k= Each Transformer decoder layer (-) has extra multi-head tross-attention layer compared to the Transformer encoder layer. The instance- aware query embeddings g are then fed into unified query representation part to generate predictions for three sub-tasks, including classification, localization and segmentation.
Unified Query Representation After Transformer decoder, each instance-un aware query embedding in g represents the features of corresponding instance. The supervision of three sub-tasks (classifi- cation, localization and segmentation) an unified form (e.g. vector) is the last piece of the puzzle to achieve parallel predictions. Fig. 2 shows the learning of UQR. We mainly describes the joint supervision of multi-task learning as well as the training and inference processes of mask branch.
In details, UQR is learned under the supervision of classification, localization and mask branches. Both classification and localization branches are the same as DETR [1]. The classification branch is fully-connected (FC) layer that predicts the class confidences è. The localization branch is multi-layer perception (MLP) with hidden size 256 and predicts 4 box coordinates b. Similar to localization branch, the mask branch also a multi-layer perception with hidden size 1024 and predicts mask vectors i e I Jxne, n& is the dimension of each mask vector. During training, the mask

---

PC
Rechs
Encoding (Training)
Class
MLP
L fgion
Box vector
gk
Compression Coding
fi
Mask vector
Decoding (Inferenee)
Mask vector label
UQR: Unified Query Representation
Figure 2: The learning of the proposed unified query representation. g refers to the learned instance-award query embeddings. é, 6, are predicted class, box and mask vectors, respectively. in is the binary masks reconstructed. m and v denote ground-truth binary masks and mask vectors, correspondingly. Laiss Lr, &gion and fivee are the losses for classification, box regression and mask segmentation, respectively.
vectors predicted are supervised by the ground-truth mask vectors E RJxx generated from spatial mask E R N by the mask compression coding described below, N is the spatial dimension of binary mask. While for the inference, the predicted mask vectors can be used to reconstruct the spatial masks n by the inverse process of compression coding. Note that each Transformer decoder layer learns such an UQR and auxiliary supervision is adopted for better performance.
Mask Compression Coding As mentioned above, predicted mask vectors i generated by the mask branch are supervised by the ground-truth mask vectors v. Here, we explore three compression coding methods transform 2D spatial binary masks into ID mask vectors, including Sparse Coding [41], Principal Component Analysis (PCA) [42] and Discrete Cosine Transform (DCT) [43].
Sparse Coding compresses the binary mask as a sparse combination of ny atoms from an overcom- plete dictionary D E R x Ground-truth mask vectors can be obtained from m through solving minimum of Lasso [52] problem:
(v" D*) argmin( (v.D)
uDIl2 B(lu|l)), s.t. ||Dell2 Ve [L, mp]
(3)
where 3 is the regular coefficient. The binary masks m can be reconstructed via m eD.
PCA transforms binary masks to low-dimensional mask vectors via matrix factorization. The process can be summarized the following optimization problem:
P = argmin||m - PPPP IP, s.I. Pp" = -Une
(4)
where E R2-2xm Xny and Un E R X are the projection matrix and unit matrix, respectively. mP meanwhile binary masks can be
Ground-truth mask vectors can be represented reconstructed by ń = uP
DCT first transforms ground-truth binary masks m into frequency domain according to f = AmAT. where IR N the transform matrix and the (h, 1) element of A can be calculated by Ap.!
1+sign(h) - cos (L+0.5) h.]l The ground-truth mask vectors v can be encoded by sampling the low- N
frequency components from f. The binary masks mn can be recovered through the inverse sampling and transformation nn A- --A where Í is the inverse sampling result from é.
Loss Function We extend object ground-truths with mask vectors as y = (c, b, v) and corresponding predictions are ÿ = (č, b, 0). The overall loss function for supervision can be expressed as:
Linst Liiet + Xvece (v, i)
(5)
where free is the mask vector loss and we use L1 loss in practice. Xnee is the corresponding weight and Latet same as Eq. 2. Note that mask loss included in bipartite matching. The participation of mask loss may affect the global matching between the object queries and ground-truths.
5

---

3.3 Discussion
Whereas our SOLQ shares similarities with MEInst [21], DCT-Mask [45] and ISTR [30] in the compression coding of spatial mask, the main differences are described as follows. SOLQ produces the instance masks together with the instance class and the location in a parallel way. It aims to learn an unified query representation for better multi-task learning. The predictions of three sub-tasks are all obtained in regression manner. Also, SOLQ is an end-to-end instance segmentation framework without any post-processes, like NMS. In contrast, DCT-Mask encodes the instance masks based on the Rol features cropped by the bounding boxes from detection branch. The mask encoding in MEInst and ISTR requires extra optimization process obtain the optimal project matrix for reconstructing spatial masks. So they are not learned in an end-to-end manner. Also, our concurrent work, QueryIne Inst [24] performs instance segmentation in an end-to-end fashion based on Sparse RCNN [31]. The overall architecture mainly follows 'detect-then-segmen paradigm. The features cropped by bounding boxes interact with the updated queries to generate the segmentation masks.
4 Experiments
4.1 Dataset and Metries
We validate our method on COCO benchmark [19]. COCO contains 115k images for training, 5k for validation and 20k for testing, involving 80 object categories with instance-level segmentation annotations. We report results on COCO 2017 rest-dev set for state-of-the-art comparison and the results on COCO 2017 val set for ablation studies. Consistent with Mask R-CNN [2], the standard COCO metrics including Aphoz AP API APp and AP AP AP API, are used to evaluate the performance of object detection and segmentation.
4.2 Implementation Details
For fast convergence, we build the SOLQ on top of Deformable DETR [35] in practice. ResNet [53]. pretrained on the ImageNet [54] is employed as the backbone and multi-scale feature maps from C3 to C6 stages are used. For the deformable attention, the number of heads is set as 8 and the number of sampling points is set as 4. For the mask branch, R& is set to 256, hidden dim of MLP is 1024 and Avee is 3.0. Following DETR, Xcta = 2, AL1 5, Agion 2. We train our model with Adam optimizer with weight decay of 1.0 x 10 10-4 Models are trained for 50 epochs with the initial learning rate 2.0 x 10-4 and decayed at 40th epoch by factor 0.1. Multi-scale training is adopted, where the shorter side is randomly chosen within [408, 800] and the longer side is less or equal to 1333. All experiments
conducted over 8 Tesla V100 GPUs with batch size 32 except the comparison in Sec. 4.4. Since the D-DETR with SQR can only be trained with batch size 16, we also perform the D-DETR with UQR under the same setting for fair comparison.
4.3 Comparison with Sfate-of-the-arts
As shown in Tab. 1, we compare SOLQ with state-of- methods on COCO test-dev set. Our method achieves best performance on both APoeg and Aploz metrics. Compared to the typical two- stage methods Mask R-CNN [2] and Cascade Mask R-CNN [7], SOLQ with ResNet-101 surpasses in Apeeg by 2. 11 and 0.9%, respectively. Besides, we also compare SOLQ with state-of-the-an one-stage methods CondInst [9] and SOLOv2 [16], which are built based on dynamic convolution. Our method outperforms them 1.8% and 1.2% in APses, respectively. Further, based on the recently fully- Transforner framework for instance segmentation and it can achieve 46.7% APseg and 56.5% Apboz, To further validate the quality of boundary prediction, we also evaluate SOLQ using the Boundary AP [55] in Appendix A.1.
introduced Swin Transformer [40], our SOLQ will serve
It is worthy noting that SOLQ performs well on objects of different scales, especially on small and middle scale objects. For example, SOLQ with ResNet-101 surpasses SOLOv2 by 5.2% in AP and 0.9% in APoeg respectively. It should be owing to the mask compression encoding of spatial binary mask. SOLOV2, the mask predictions are supervised by the ground-truth mask of 1/4 instance size the performance of small objects are not optimized well. For our SOLQ, the mask

---

Table 1: State-of-the-an art comparison the COCO 2017 test-dev set. All the models trained with multi-scale and tested with single scale. " and *** are the results reported in ISTR [24] and QueryInst [24], respectively. For experiments with Swin-L backbone [40], the shorter side of input is randomly chosen within [400, 1200] and the longer side is less or equal to 1536.
Method
Backbone
Epochs
AP
APs
APM
AP
Ap
S
APM
APL.
Mask R- R-CNNI [2]
R50-FPN
36
37.5
21.1
39.6
48.3
41.3
24.2
43.6
51.7
Cascade Mask R-CNN" [7]
R50-FPN
36
38.6
21.7
40.8
49.6
44,5
HTC* [4]
R50-FPN
36
39.7
22.6
42.2
50.6
44,9
MEInst [21]
R50-FPN
36
33.5
19.3
35.7
42.1
42.5
25.6
45.1
52.2
CondInst [9]
R50-FPN
36
37.8
21.0
40.3
48.7
42.1
25.1
44.5
52.1
BlendMask [22]
R50-FPN
36
37.8
18.8
40,9
53.6
43.0
25.3
45.4
54,0
SOLOv2 [16]
R50-FPN
72
38.8
16.5
41.7
56.2
40.4
20.5
44.2
53.9
QneryInst [24]
R50-FPN
36
40.6
23.4
42.5
52.8
45.6
ISTR [30]
R50-FPN
36
38.6
22.1
40.4
50.6
46.8
27.8
48.7
59.9
SOLQ, ours
R50
50
39.7
21.5
42.5
53.1
47.8
27.6
50.9
61.6
Mask R-CNNT [2]
R101-FPN
36
38.8
21.8
41.4
50.5
43.1
25.1
46.0
54.3
Cascade Mask R-CNN [7]
R101-FPN
36
40.0
22.5
42.5
51.2
46.2
HTC* [4]
R101-FPN
36
40.8
23.0
43.5
52.6
46.3
MEInst [2]]
R101-FPN
36
35.3
20.4
37.8
44.5
44.5
26.8
47.3
54.9
CondInst
R101-FPN
36
39.1
21.5
41.7
50.9
43.5
25.8
46.0
54.1
BlendMask [22]
R101-FPN
36
39.6
22.4
42.2
51.4
44.7
26.6
47.5
55.6
DCT-Mask [45]
R101-FPN
36
40.1
22.7
42.7
51.8
SOLOv2 [16]
R101-FPN
72
39.7
17.3
42.9
57.4
42.6
22.3
46.7
56.3
QueryInst [24]
R101-FPN
36
42.8
24.6
45.0
55.5
48.1
ISTR [30]
R101-FPN
36
39.9
22.8
41.9
52.3
48.1
28.7
50.4
61.5
SOLQ, owrs
R101
50
40.9
22.5
43.8
54.6
48.7
28.6
51.7
63.1
QueryInst [24]
Swin-I.
50
49.1
31.5
51.8
63.2
56.1
SOLQ, ours
Swin-L
50
46.7
29.2
50.1
60.9
56.5
37.6
60.0
70.6
compression encoding encodes the high-resolution binary mask (e.g. 128 x 128) into low-dimension mask vectors using the sparsity characteristic of the binary mask and keeps the principle information.
4.4 UQR vs. SQR
In this section, we show the comparison between the separate query representation (SQR) (see Fig. ((a)) DETR and the UQR used in SOLQ (see Fig. 1(b)). As shown Tab. we surprisingly find that UQR in SOLQ can boost the detection performance of DETR with great margin, improving Apbox by 2.3% and 2.0% with ResNet50 and Res While in comparison, separate query representation (SQR) only improve the Ap by 0.1% and the segmentation performance with 33.4 APseg is much lower than that UQR. The large improvement in Aphor shows the effectiveness of our proposed UQR, owing to the unified learning of query representation. For efficiency comparison between them, please refer to the Appendix A.2.
We also report the detection performance of both Faster R-CNN and Mask R-CNN since Mask R-CNN is built on top of Faster R-CNN. Compared Faster R-CNN, Mask R-CNN improves Ap by 0.6% and 1.1% with ResNet50-FPN and ResNet101-FPN, respectively. As we see, multi-task learning tends to improve the performance with each other and the performance can be further improved if these sub-tasks are learned using an unified representation. SOLQ learns the UQR to perform classification, localization and segmentation simultaneously in regression manner. For both SQR and Mask R-CNN, full-connected layers are employed to classify the objects and regress box coordinates while the mask generated by full convolution network is supervised by 2D spatial mask.
Further, Fig. shows the visualization comparison between the DETR with SQR and our SOLQ. Over- all, SOLQ generates much more fine- fine-grained masks and provides better object detection performance. We also show some failure cases in occluded environments (see Appendix A.5).
4.5 Ablation Studies
In this section, we also ablate several critical factors in SOLQ by progressively adjusting each factor in the system. It can be seen that each factor contributes the final success of the SOLQ. Note that
7

---

Table 2: Comparisons between Unified Query Representation (UQR) and Separate Query Represen- tation (SQR) on the COCO 2017 val set. D-DETR denotes Defoemable DETR and D-DETR refers reimplementment version. D-DETR" with SQR means that add an extra FPN-style branch as shown in Fig. (a) to perform instance segmentation on top of D-DETR*. 'f" and 't' are the results
reported in ISTR [24] and Sparse RCNN [24], respectively.
Method
Backbone
Epochs
APre
AP
AP
APhan
APlon
Faster RCNN [6]
R50-FPN
36
40.2
24.2
43.5
52.0
Mask R-CNNÍ [2]
R50-FPN
36
37.0
40.8 (+0.6)
24.0
44.4
52.9
D-DETR [35]
R50
50
45.4
26.8
48.3
61.7
D-DETR-
R50
50
45.5
27.3
48.7
62.0
D-DETR* +SQR
R50
50
32.2
45.6 (+0.1)
27.2
48.8
61.5
R50
50
39.5 (+7.3)
47.8 (+2.3)
28.7
51.1
63.7
Faster RCNNÍ [6]
RI01-FPN
36
42.0
26.6
45.4
53.4
Mask R-CNNI [2]
R101-FPN
36
38.8
43.1 (+1.1)
25.1
46.0
54.3
D-DETR
RTOI
46.3
62.3
D-DETR*
RIO1
33.4
(+0.1)
28.1
62.2
D-DETR'+UU +UQR
RI01
40.2 (+6.8)
(+2.0)
29.9
52.1
64.6
D-DETR -QQ
SOL
Figure 3: Visualization comparisons between Unified Query Representation (UQR) and Separate Query Representation (SQR) on the COCO 2017 val set. For more visualization comparison, please refer Appendix A.4.
our ablation experiments are conducted using the single feature level with C5 and validated on the COCO 2017 val set.
Mask Compression Coding Methods We compare the impact of different compression coding methods to encode binary masks. As shown in Tab. 3a, flatting the spatial masks into ID mask vectors directly for supervision can obtain decent segmentation result while improving Apboz by 1%. DCT produce the best performance in both Apsea and Aphoz metrics. The main reason is that the loss of mask compression caused by DCT is relatively small, compared to the Sparse Coding and PCA methods. Also, DCT can be employed for each image in online manner while Sparse Coding and PCA are performed on the whole training set to get "principal components" or "dictionary" in an offline way. Therefore, choose DCT our default method for mask compression coding.
Mask Vector Loss Weight Tab. 3b shows the effect of adjusting the weight of mask vector loss. When the weight of mask vector loss Avec is set 3.0, SOLQ gets the best performance in both AP"eg and Ap The regression dimension of DCT vector (e.g. 256) is usually larger than that of the bounding box (e.g. 4), so the magnitude of mask vector loss is larger than that of the regression loss. Therefore, an appropriate Avec can keep the balance between the mask branch and localization branch such that these two branches can be jointly optimized better.
Mask Vector Loss Stage We also ablate impact of the number of decoder stages enabling mask vector loss in Tab. 3c. It can be seen that adding auxiliary mask vector loss on all decoders can improve by 2.9% and 1. 1% on Ap=eg and Apboz, respectively. The experimental results show that auxiliary loss employed in multiple decoders is helpful to both mask and detection branches. Note

---

that auxiliary loss of detection branch is enabled in all decoder stages in this ablation, so the gain of detection branch not as large as the gain of the mask branch.
Spatial Resolution of Binary Mask As mentioned above, mask compression coding projects the 2D N x N ground-truth binary masks into ID mask vectors for supervision. In Tab. 3d, explore the impact of the spatial resolution of ground truth binary mask. The segmentation performance Ap is improved from 30.8 to 32.6 N increases from 64 to 128 and AP no longer improves when N is greater than 128. Ground-truth binary mask with high resolution can keep more instance details but needs high-dimension mask vectors reconstruct. Therefore, once the number of coefficients ny is chosen, N has the most suitable value corresponding to 7.k-
Dimension of Mask Vector Similar to the spatial resolution of ground truth binary mask, there is also a suitable value ny for given N. Tab. 3e shows that ni 256 achieves the best performance in both Apce9 and Apboe when the spatial resolution N is set 128. Theoretically, mask vector with higher dimension should reconstruct a better binary mask. However, high-dimension mask vector will enlarge the regression dimension, making it hard optimize the mask branch. As a result, the quality of the binary mask reconstructed reduces lot.
Table Ablation studies validated on the COCO 2017 val set. All experiments use the single feature level with C5 in ResNet50.
(a) Impact of different binary mask encoding-decoding methods. Flatten means that reshape the 2D binary masks (28x28) into ID mask vectors (784) directly. then optimize with L2 dice loss jointly.
Type
Aprng
A
AP
AP
Aphos S
APhuz
Aphoz
det
39.4
20.6
43.0
55.5
Flatten
29.7
11.6
33.4
49.3
40.4
20.2
44.7
58.8
Sparse Coding
11.3
6.2
12.0
17.5
40.6
20.6
44.6
59.3
PCA
31.6
12.5
35.4
52.7
41.0
20.5
45.2
60.0
DCT
32.6
12.8
37.1
54.5
41.3
20.7
45.4
60.1
(b) Affect of adjusting mask vector loss weight. Mask vector loss enahled only the decnder layer, nu = 256. Spatial resolution of ground truth
(c) Ablation of the number of decoder stages abling mask vector loss. For example, when stage is 4. it means that enable the last decoder layer
binary mask N
128.
with mask vector loss and Anee
3. n&
256.
Avec
AP
AP
AP
Ap
AP
AP
Num. Ap
AP
AP
Ap
AP5
AP
0.3
25.2
$1.0
22.2
39.3
59.9
41.8
29.7
53.5
27.7
40.2
60.4
0.7
26.7
52.1
24.4
39.6
60.1
42.5
31.8
55.4
32.1
40.8
43.3
27.2
53.4
24.6
39.7
61.3
43,3
32.0
55.2
32.4
40.7
61.0
42.9
2
28.9
53.5
40.0
60.4
42.8
32.4
55.7
33.1
41.0
61.2
43,5
29.7
53.5
28.4
40.2
60.4
42.8
32.3
55.3
33.1
40.9
60.9
43,4
29.6
52.3
25.6
40.2
60.1
42.4
32.6
55.9
33.4
41.3
61.7
43.4
(d) Effect of the spatial resolution of ground truth binary mask. Mask vector loss is enabled all
(e) Impact of the dimension of mask vector. Mask vector loss is enabled in all decoder layers. Avec
decnder layers and Avec = 3, 72g 256.
3 and spatial resolution of binary mask 128.
N
AP
AP
AP
Aplox
APh
APlon
R2g
Ap*gg
AP
AP
Aplos
APlaz
A Plazer
64
30.8
54.8
30.8
40.5
61.2
42.7
144
31.5
55.6
31.6
40,9
61.3
43.4
96
31.5
55.4
31.8
40.7
61.4
43.1
256
32.6
55.9
33,4
41.3
61.7
43,4
128
32.6
55.9
33.4
41.3
61.7
43.4
300
31.0
54.7
30.9
40.4
60.8
42.8
256
32.3
55.4
32.8
40.6
60.7
43.3
400
30.9
55.1
30.2
40.6
61.3
43.0
Conclusion
In this paper, we present SOLQ, a new instance segmentation framework. Based on DETR, SOLQ learns an unified query representation, which is used to predict the instance masks parallel with object detection in an end-to-end manner. To make the mask representation consistent with the query embedding, SOLQ projects the high-resolution spatial masks into low-dimensional mask vectors and regards mask prediction as the regression of mask vectors. SOLQ is truly one shot framework without any two-stage operations, like ROIAlign. On the challenging COCO dataset, instance segmentation task and greatly improves the
SOLQ achieves state-of-the-an performance detection performance DETR thanks to the multi-task learning. We believe that SÓLQ will serve strong baseline for instance segmentation for its excellent performance and simplicity.

---

References
[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. Eur. Conf. Comp. Vis., 2020.
[2] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cna. In Proc. IEEE Int. Conf.
Comp. Vis., 2017.
[3] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
[4] Kai Chen, Jiangmiao Pang, Jiaqi Wang. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jianping Shi, Wanli Ouyang. et al. Hybrid task cascade for instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
[5] Shu Liu, La Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmenta- tion. In Proc. IEEF Conf. Comp. Vis. Patt. Recogn. 2018.
[6] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. Proc. Advances in Neural Inf. Process. Syst. 2015.
[7] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018.
[8] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proc. IEEE Int. Conf. Comp. Vis., 2019.
191 Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. Proc. Eur. Conf. Comp. Vis., 2020.
[10] Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor-free instance segmentation. Proc. IEFE Conf. Comp. Vis. Patt. Recogn., 2020.
[11] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou. Deep snake for real-time instance segmentation. Proc. IEEE Conf. Comp. Vis. Putt. Recogn., 2020.
[12] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proc. IEEE Int. Conf. Comp. Vis., 2019.
[13] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proc. IEEE Int. Conf. Comp. Vis., 2019.
[14] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue Van Gool. Dynamic filter networks. Proc. Advances in Neural Inf. Process. Syst, 2016.
[15] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations.
In Proc. Eur. Conf. Comp. Vis., 2020.
[16] Xinlong Wang. Rufeng Zhang, Tao Kong. Lei Li, and Chunhua Shen. Solov2: Dynamic, faster and stronger. Proc. Advances in Neural Inf. Process. Syst., 2020.
(17] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proc. IEEE Conf. Comp. Vis. Pati. Recogn., 2017.
[18] Lingling Tao, Fatih Porikli, and René Vidal. Sparse dictionaries for semantic segmentation. In Proc. Eur. Conf. Comp. Vis., 2014.
[19] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. Eur. Conf. Comp. Vis., 2014.
[20] Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-aware aware semantic segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2017.
[21] Rufeng Zhang. Zhi Tian, Chunhua Shen, Mingyu You, and Youliang Yan. Mask encoding for single shot instance segmentation. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
[22] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan. Blendmask: Top-down meets bottom-up for instance segmentation. Proc. IEEE Conf. Comp. Vis. Patt. Recogn.. 2020.
[23] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár. Tensormask: foundation for dense object segmentation. Proc. IEEE Int. Conf. Comp. Vis., 2019.
[24] Yuxin Fang. Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
[25] Naiyu Gao, Yanhu Shan, Yupei Wang Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
10

---

[26]
Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. Sgn: Sequential grouping networks for instance
segmentation. In Proc. IEEE Int. Conf. Comp. Vis., 2017.
[27] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv:/708.022:
2017.
[28]
Alejandro Newell, Zhiao Huang. and Jia Deng. Associative embedding: and-to-end learning for joint detection and grouping. arXiv:1611.05422 2016.
[29] Enze Xie. Peize Sun, Xiaoge Song. Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In Proc. IEEE Conf. Comp. Vis. Putt. Recogn., 2020.
[30] Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Istr: End-to-end instance segmentation with transformers. arXiv preprint 2021.
[31] Peize Sun, Rufeng Zhang. Yi Jiang. Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end ohject detection with learnable proposals. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Inf. Process. Syst., 2017.
[33] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2018
[34] Zilong Huang, Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu, Cenet: Criss-cross attention for semantic segmentation. In Proc. IEEE Int. Conf. Comp. Vis., 2019.
[35] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to end object detection. arXiv:2010.04159, 2020.
[36] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv:20/2.00364, 2020.
[37] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv:2010. /929, 2020.
[38] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou, Training data-efficient image transformers & distillation through attention. arXiv:2012.1287, 2020.
[39] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. wxXi:2202.122122 2021.
[40] 7 Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arxiv:2 14030, 2021.
[41] David L Donoho. Compressed sensing, Proc. IEEE. Trans. Inf. The., 2006.
[42] Svante Wold, Kim Eshensen, and Paul Geladi. Principal component analysis. Proc. Chem. Int. Lab. Syst., 1987.
[43] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. Proc. IEEE. Trans. Comp.. 1974.
[44] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In Proc. JEEE Conf. Comp. Vis. Putt. Recogn., 2020.
[45] Xing Shen, Jirui Yang, Chunbo Wei, Bing Deng, Jianqiang Huang, Xiansheng Hua, Xiaoliang Cheng, and Kewei Liang. Det-mask: Discrete cosine transform mask representation for instance segmentation.
2XXiv:2011.000 1.09866, 2020.
[46] Shao- Yuan Lo and Hsueh-Ming Hang. Exploring semantic segmentation on the det representation. In ACM Int Conf. Mul, Med. Asia. 2019.
[47] Harold W Kuhn. The hungarian method for the assignment problem. Nav Rex. Log. Qua., 1955.
[48] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proc. IEEE Int. Conf. Comp. Vis., 2017.
[49] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2019.
11

---

[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2016.
[51] Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2010.
[52] Robert Tibshirani. Regression shrinkage and selection
lasso. Jour. Roy. Stat. Soc., 1996. [53] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2016.
[54] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. Jour. Comp. Vis., 2015.
[55] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation, In Proc. IEEE Conf. Comp. Vis. Patt, Recogn.. 2021.
12


Reviews:
[{"review_id": "tVDy42JAYKW", "reviewer": "Reviewer_xtMD", "summary": "The paper proposes a different approach to instance segmentation using DETR-like\narchitecture. Instead of the FPN-based approach adopted in the original DETR, they opt for\nregressing directly an encoded version of the mask.\n\nThis approach allows to perform segmentation in a cheaper manner, and also allows to use the\nsegmentation objective in the auxiliary loss (not possible in original DETR due to\nprohibitive cost), which gives some substantial improvement.\n\nThe results shown on Coco are competitive with current state of the art.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is well motivated, and the paper is easy to follow. The ablations thoroughly\nexplore the technical choices being made (although I have some questions about the said\nablations, see below).\n\nOverall, I find the contribution valuable for the community, since efficient and accurate\ninstance segmentation is largely unexplored for DETR-like models. I thus lean to accept this\npaper. In the rest of this review, I list some questions and possible ways to make the\nsubmission even stronger.\n\nNote that [4] also tackled instance segmentation with a DETR-like architecture (although\nmainly focusing of video), and should be discussed in the related work section.\n\n## Comparison with state-of-the-art - SWIN results\n\nI am a bit confused by the presence of the SWIN-L backbone in the main table, without much\nexplanations.\n\nWhile it is interesting in theory to combine DETR and SWIN (that creates the first\n\"transformer only\" detector that I am aware of), proper comparison should be made with other\nsystems. The SWIN paper itself provides numerous detection results, including a 51.1 mask\nAP, much better than what is being reported in this paper. It is to be noted that direct\ncomparison is a bit difficult since this particular result involves numerous bells and whistles. I\ndo think, however, that a proper comparison, on comparable settings, is advisable for\ndetection results using this backbone.\n\n\n## Ablations consistency\n\nFrom table 3.b, it appears that the optimal coefficient for the $\\lambda_\\text{vec}$\nparameter is 3.0. However, ablations in 3.c, 3.d and 3.e are all conducted with $\\lambda_\\text{vec}=0.3$\n\nWhat is the reason behind this? I am not sure what to conclude from ablations performed with\nsub-optimal hyper-parameters.\n\n## Timings\n\nTraining time could be reported for completeness.\n\nMore importantly, I believe it would be valuable to provide inference timings for this\nmodel, compared to the competing ones. One of the major claim of the paper is the simplicity\n(one-shot, no ROIalign, ...), but it is not clear that the simplicity actually translates to\nfaster inference.\nIn particular, it is not clear to me how costly the DCT decoding actually is.\nSimilarly, if possible a FLOPS analysis would be interesting.\n\n\n## Mask quality\nRecent papers such as [1] have argued that Mask-IoU doesn't capture the full picture in the\nquality of the masks. For example, despite reaching fairly good AP, it is known that\nMask-RCNN masks are sometimes \"blobby\". It would be interesting to provide more qualitative\ncomparisons of the performance of the model in various situations. Fig. 4 partially answers\nthat question by showing failure modes in occluded objects, but what about, for example,\nhigh frequency details in the mask boundaries? (eg the individual fingers of a hand) Would\nthey be captured adequately? \n\nBeyond qualitative results, it would also be interesting to provide quantitative metrics,\nsuch as the Boundary-IoU [1], and compare with recent high-quality mask prediction models\nsuch as PointRend [2] and BMask R-CNN[3]\n\n## Segmentation loss in matching?\n\nFrom the equations provided in section 3.1 and 3.2, it would seem that the segmentation loss\nisn't used as part of the matching. Is this correct?\nIf so, have you also tried incorporating it in the matching cost? Intuitively, it is\ndesirable to make the loss and the matching as similar as possible, to ensure a smoother\noptimization.\n\n\n## Panoptic segmentation?\n\nIn the original DETR paper, one of the main application of the segmentation head was\nPanoptic segmentation. How easy would it be to apply this method to this setting as well? In\nparticular, does the mask encoding work well for stuff segments? Would it be easy to merge\nthe predictions from different masks? (similar to the pixel-wise argmax done in DETR).\n\n\n\n[1] Boundary IoU: Improving Object-Centric Image Segmentation Evaluation, Cheng et al\n\n[2] PointRend: Image segmentation as rendering, Kirillov et al\n\n[3] Boundary-preserving Mask R-CNN, Cheng et al\n\n[4] End-to-End Video Instance Segmentation with Transformers, Wang et al\n\n# Post rebuttal\nIn light of the rebuttal, I’m increasing my score (6->7)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing stands out.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "rUm4nadQsnf", "reviewer": "Reviewer_AEBz", "summary": "The paper proposes and end-to-end instance segmentation approach based on recently proposed Transformer-based object detector DETR. In DETR instance segmentation is done by upsampling of ResNet features from multiple stages, which is computationally expensive. Instead, the authors propose to predict instance masks together with bounding box and class labels, by compressing the masks. This results in a simple and efficient detection and instance segmentation model. Moreover, AP on both tasks improves significantly over DETR and Deformable DETR, achieving very impressive results on par with state-of-the-art, on the challenging COCO dataset. The paper also provides a detailed ablation study of the proposed model.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the paper is based on an original and novel idea, predicting compressed instance masks directly with DETR model, avoiding complex operations like proposal generation, non-maximum suppression and ROIAlign, compared to Mask R-CNN.\n\nQuality: the proposed approach to instance segmentation is simple and elegant. The experimental evaluation fully supports the claims, and there is a detailed ablation study for hyperparameter and representation choices. The performance of the proposed approach on COCO benchmarks is very impressive. Similarly to Mask R-CNN Vs Faster R-CNN, adding mask supervision improves bounding box prediction performance.\n\nClarity: the paper is very well written and is easy to follow. I have only a few questions/suggestions to improve the paper:\n- attention visualizations in DETR show that decoder attends to object extremities in order to predict bounding box. It would be interesing to see SOLQ decoder attention visualizations, if it attends to object outline?\n- I found \"vector loss\" confusing, would \"Mask vector loss\" be a better name?\n- would be great to have more details on the Swin-SOLQ experiments, is it \"almost\" CNN-free object detector?\n- I did not find if L_vec loss is also used in cost for bipartite matching?\n\nSignificance: the paper greatly improves over DETR and Deformable in detection and segmentation performance, and computational complexity. I expect it to be become a new baseline for both tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations and potential negative societal impact of their work adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "kt7I-rQ71Fl", "reviewer": "Reviewer_f92m", "summary": "This paper proposes SOLQ, which combines Deformable DETR (D-DETR) and DCT for instance segmentation. The experiments verify the effectiveness of SOLQ to a certain extent.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- *Originality & Novelty:* **The novelty of SOLQ is limited.** SOLQ studies PCA and DCT for mask representation, which has already been studied in MEInst and DCT-Mask. SOLQ simply verifies DCT for mask representation again on the recently proposed D-DETR.\n- *Quality:* \n  - **The motivation of using sparse coding for instance segmentation is not verified in the experiment**. As stated in lines `41-43` of the paper (also in MEInst), the motivation of using sparse coding (e.g., PCA or DCT) for mask representation is that \"spatial mask labels results in high computation cost, and sparse coding results in more compact mask representation\". But the efficiency of sparse coding (e.g., number of parameters, theoretical FLOPs, and most importantly, FPS) is not studied in the experiments. The paper should provide some quantitative results to explain the reason to use sparse coding for instance segmentation.\n  - **The spatial mask representation baseline for instance segmentation (SQR) is relatively weak.** The paper uses Separate Query Representation (SQR), which is originally proposed by DETR, as a kind of spatial mask representation baseline for instance segmentation in D-DETR framework. But SQR is a raw & straightforward design for panoptic segmentation other than instance segmentation, and the results are relatively weak. Since D-DETR / SOLQ has bounding box predictions, I suggest using the box prediction to pool RoIs for mask predictions in an interleaved manner just like HTC as a spatial mask representation baseline. Or the paper can choose to make **fair** comparisons with other arts based on spatial mask representation to verify the effectiveness and efficiency of sparse coding (e.g., PCA or DCT) for mask representation.\n  - **The instance segmentation performance of using DCT is not satisfactory**. At first glance the numbers reported in the paper are OK. But the performance gap between $\\mathrm{AP}^{box}$ and $\\mathrm{AP}^{mask}$ is too **big**, i.e., about $8$ $\\mathrm{AP}$, while the gap for the Mask R-CNN family is usually less than $6$ $AP$. This indicates that this kind of mask representation is not satisfactory. The paper claims that SOLQ can improve the object detection performance, so do Mask R-CNN family and many other instance segmentation approaches to my knowledge, since mask annotations can provide additional supervision for object detection. Note that under similar object detection performance, the instance segmentation performance of SOLQ is not competitive. An HTC with $46.3$ $\\mathrm{AP}^{box}$ is $1.1$ $\\mathrm{AP}^{mask}$ higher than SOLQ with $47.8$ $\\mathrm{AP}^{box}$. SOLQ needs to exploit a much better object detector for a reasonable instance segmentation performance. \n  - **Is SOLQ end-to-end?** I think the ultimate objective of building an end-to-end system is to remove the need for many hand-designed components and heuristics. While the use of DCT in SOLQ **introducing two extra hyper-parameters**, i.e., the spatial resolution of ground truth binary mask $N$, and the dimension of mask vector $n_k$. From the ablation study in Table 3 (d) & (e), the performance of SOLQ is **very sensitive to** these two hyper-parameters. Therefore the approach adopted by SOLQ is **clearly inconsistent with** the motivation & goal of designing an end-to-end system. On the other hand, the elimination of NMS is directly inherited from the DETR family, which shouldn't be considered as a contribution of SOLQ.\n- Clarity:\n  - **This paper makes many unfair comparisons in the experiments.** As a cascade / iterative framework, the qualified competitors for SOLQ are not one-stage MEInst & CondInst & SOLO, etc. But Cascade Mask R-CNN and HTC, etc. However, SOLQ does not make fair comparisons with most approaches in Table 1: SOLQ uses a longer training schedule. \n  - **Figure 1 is very misleading**: the block of \"DConv\" has a similar size as the Res Blocks, $\\mathcal{F}$ & $\\mathcal{H}$ (they both have several Transformer layers), which makes the FPN-style instance segmentation design (actually, it is a panoptic segmentation design) of DETR seems to be very heavy. I think the paper should report FPS and compare the efficiency with other arts **quantitatively** other than using these bells and whistles.\n  - The experiments are conducted on D-DETR, but in lines `10-12` & lines `72-75` the paper claims they observe improvements on **original DETR**, which is not verified in the experiments. Also, the abstract, introduction & method parts are built on original DETR, while the experiments switch to D-DETR. Therefore many details about the setup of the D-DETR framework are missing.\n- Significance: Overall, I believe SOLQ is just yet another instance segmentation model without significant contribution to the object- or instance-level recognition tasks & model design methodology.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N / A", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "UT9jrzx10ck", "reviewer": "Reviewer_94Kv", "summary": "This paper tries to combine DETR and Mask Encoding techniques for end-to-end instance segmentation: DETR could directly separate different objects and Mask Encoding could encode the arbitrary instance mask into a low-dimensional and fix representation. To verify the effectiveness of each component, the authors conducted extensive ablations for each component. The final performance is inspiring, espically using Trasnformer backbone.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "== Pros ==\n\n1. The paper is well motivated and shows a reasonable story. The wring is good and easy to follow.\n2. Using Mask Encoding to encode the mask is a good choice for modeling the mask, and the performance (especially AP_s) has been largely improved. \n3. The UQR is better than SQR, and it could further improve the object detection performance.\n\n== Cons == \n\n1. My main concern is originality. From my understanding, Mask Encoding/Compression has been explored by previous work (i.e., DCT-Mask, MEInst). So adopting it into DETR is straightforward. It could only demonstrate that Mask Encoding works well for DETR-style architecture, but both techniques are not your contribution. \n2. I agree that no previous work has tried combing these techniques together to form an end-to-end instance segmentation system. The performance is also great. But combining weakened your contribution.\n3. In L207, the authors claim that DCT-Mask is not learned in an end-to-end manner. However, I believe DCT-Mask is in fact an end-to-end trainable system (please refer to Sec.3 of their paper).\n4. SOLQ gets higher AP on small objects, however, it suffers from large objects compared with SOLOv2 (Table 1). Could you explain why?\n5. What is the difference between UQR and SQR? From my understanding, UQR is using the encoded mask for supervision, while SQR is using a binary mask. \n\n== Overall ==\n\nIncluding Mask Encoding into DETR could directly enable DETR to do instance segmentation. That is good. However, Mask Encoding has been used in the previous two-stage (DCT-Mask), and one-stage (MEInst) methods.  \nIf you just change the detector from Faster R-CNN/FCOS to DETR, the novelty would be really incremental. It is\nimportant to discuss the differences and highlight the novelty.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a direct way to do instance segmentation in DETR: instead of producing segmentation masks with an FPN, it regresses segmentation masks from compressed representations as queries. This allows training DETR for segmentation end-to-end (while the original DETR and follow-ups have segmentation trained as a second step to detection). 

This paper is roughly the result of the combination of (DF-)DETR with mask encoding (DCT), which is criticized by reviewer 94Kv and f92m. I believe there is nonetheless originality in making it work so well that the approach reaches very strong numbers in segmentation on COCO (39.7 APseg with a ResNet-50 backbone and 45.9 with a Swin-L). 

Reviewer f92m constructively points out that the gap between APbox and APseg is bigger for this approach than for others, which could mean that it requires a stronger detector. Another interpretation can be that training the DF-DETR model with this additional segmentation loss end-to-end boosts the APbox.

Overall, I believe the paper presents a significant contribution, and the authors answered some of the concerns of the reviewers. This is suitable for publication at NeurIPS.
Decision: Accept (Poster)