PatchDCT: Patch Refinement for High Quality Instance Segmentation
Authors: Qinrou Wen
Jirui Yang
Xue Yang
Kewei Liang
Venue: ICLR 2023 poster (2023)
Keywords: Instance Segmentation

Abstract: High-quality instance segmentation has shown emerging importance in computer vision. Without any refinement, DCT-Mask directly generates high-resolution masks by compressed vectors. To further refine masks obtained by compressed vectors, we propose for the first time a compressed vector based multi-stage refinement framework.  However, the vanilla combination does not bring significant gains, because changes in some elements of the DCT vector will affect the prediction of the entire mask. Thus, we propose a simple and novel method named PatchDCT, which separates the mask decoded from a DCT vector into several patches and refines each patch by the designed classifier and regressor. Specifically, the classifier is used to distinguish mixed patches from all patches, and to correct previously mispredicted foreground and background patches. In contrast, the regressor is used for DCT vector prediction of mixed patches, further refining the segmentation quality at boundary locations. Experiments on COCO show that our method achieves 2.0\%, 3.2\%, 4.5\% AP and 3.4\%, 5.3\%, 7.0\% Boundary AP improvements over Mask-RCNN on COCO, LVIS, and Cityscapes, respectively. It also surpasses DCT-Mask by 0.7\%, 1.1\%, 1.3\% AP and 0.9\%, 1.7\%, 4.2\% Boundary AP on COCO, LVIS and Cityscapes. Besides, the performance of PatchDCT is also competitive with other state-of-the-art methods.

Published as a conference paper at ICLR 2023
PATCHDCT: PATCH REFINEMENT FOR HIGH QUALITY INSTANCE SEGMENTATION
Qinrou Wen', Jirui Yang", Xue Yang", Kewei Liang'*
Schooo of Mathematical Sciences, Zhejiang University Allibaaa Group
Department of CSE, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
(einrou.went matikw]@2a
edu. en, jirai.yjr@alibaba www.llaaaaaannaaaaaaeeee com
yangxue-2011 2019-0011200000 aa--sttuu jtuusjju. edu.
PyTorch Code: https: https://github.com http://www......oooaaaa//////aaa
ABSTRACT
High-quality instance segmentation has shown emerging importance in computer vision. Without any refinement, DCT-Mask directly generates high-resolution masks by compressed vectors. To further refine masks obtained by compressed vectors, we propose for the first time a compressed vector based multi-stage re- finement framework. However, the vanilla combination does not bring significant gains, because changes in some elements of the DCT vector will affect the pre- diction of the entire mask. Thus, we propose a simple and novel method named PatchDCT, which separates the mask decoded from a DCT vector into several patches and refines each patch by the designed classifier and regressor. Specifi- cally, the classifier is used to distinguish mixed patches from all patches, and to correct previously mispredicted foreground and background patches, In contrast, the regressor is used for DCT vector prediction of mixed patches, further refin- ing the segmentation quality at boundary locations. Experiments on COCO show that our method achieves 2.0%, 3.2%, 4.5% AP and 3.4%, 5.3%, 7.0% Boundary AP improvements over Mask-RCNN on COCO, LVIS, and Cityscapes, respec- tively. It also surpasses DCT-Mask by 0.7%. 1.1%, 1.3% AP and 0.9%. 1.7%. 4.2% Boundary AP on COCO, LVIS and Cityscapes. Besides, the performance of PatchDCT also competitive with other state-of-the-ar methods.
INTRODUCTION
Instance segmentation (Li et al., 2017; He et al., 2017) is a fundamental but challenging task in computer vision, which aims to locate objects in images and precisely segment each instance. The mainstream instance segmentation methods follow Mask-RCNN (He et al., 2017) paradigm, which often segment instances a low-resolution grid (Kang et al., 2020; Cheng et al., 2020c; Chen et al., 2019; Ke et al., 2021). However, limited by the coarse mask representation ( (i.e. 28 x 28 in to the loss of details. DCT-Mask (Shen et al., 2021) achieves considerable performance gain by predicting an informative 300-dimensional Discrete Cosine Transform (DCT) (Ahmed et al., 1974) vector compressed from 128 x 128 mask. To further improve the segmentation results of DCT- Mask, we follow the refine mechanism (Ke et al., 2022; Zhang et al., 2021; Kirillov et al., 2020) to correct the mask details in a multi-stage manner.
most of these algorithms cannot obtain high-quality segmentation results due
A straightforward implementation is to refine the 300-dimensional DCT vector multiple times. How- ever, experimental results show that this naive implementation does succeed, which improves mask average precision (mAP) by 0.1% from 36.5% to 36.6% on COCO val set. The main reason for the limited improvement is that the full 300-dimensional DCT vector is not suitable for refining some important local regions, such as wrong predicted regions and boundary regions in masks. As each pixel value the mask is calculated by all elements of the DCT vector in the inference stage, once some elements in the DCT vector change, the entire mask will change, and even the correct segmentation areas may be affected, refer to Figure la.
"Corresponding author is Kewei Liang.

---

Published as a conference paper at ICLR 2023
origin
changes
origin
changes
decode
decode
300
(a) Influence of element change in DCT-Mask
(b) Influence of element change in PatchDCT
Figure I: (a) Influence of elements changes in DCT vectors for DCT-Mask. The blue block denotes the changed elements. The box with a blue border represents the part of the mask affected by the changes in element values. The change of some elements will affect the entire mask. (b) Influence of elements changes in DCT vectors for PatchDCT. Changing some elements of vector will only affect the corresponding patch.
To overcome the above issue, we propose novel method, called PatchDCT, which divides the mask decoded from a DCT vector into several independent patches and refines each patch with a three- class classifier and regressor, respectively. In detail, each patch is first classified into one of three categories: foreground, background, and mixed by the classifier, and then previously mispredicted foreground and background patches will be corrected. Mixed patches fed into the regressor to predict their corresponding re-limensional (n < 300) DCT vectors. In the inference stage, we use Inverse Discrete Cosine Transform (IDCT) to decode the predicted vectors of the mixed patches as their refined masks, and merge them with the masks of other foreground and background patches obtain high-resolution mask. the element change of a DCT vector will only affect the corresponding mixed patch, as shown in Figure lb. In general, patching allows the model to focus on the refinement of local regions, therehy continuously improving the quality of segmentation, resulting in significant performance improvements. Our main contributions are:
also worth emphasizing that each patch is independent, so
1) To our best knowledge, PatchDCT is the first compressed vector based multi-stage refinement detector to predict high-quality masks.
2) PatchDCT innovatively adopts the patching technique, which successfully allows the model to focus on the refinement of important local regions, fully exploiting the advantages multi-stage refinement and high-resolution information compression.
3) Compared to Mask RCNN, PatchDCT improves about 2.0% AP and 3.4% Boundary AP on COCO, 3.2% AP and 5.3% Boundary AP LVIS*1, 4.5% AP and 7.0% Boundary AP on Cityscapes. It also achieves 0.7% AP and 0.9% Boundary AP on COCO, 1.1% AP and 1.7% Bound- ary AP on LVIS", 1.3% AP and 4.2% Boundary AP Cityscapes DCT-Mask.
4) Demonstrated by experiments on COCO test-dev, the performance of PatchDCT is also competi- tive with other state-of-the-art methods.
RELATED WORK
Instance segmentation. Instance segmentation assigns pixel-level mask each instance of in- terest. Mask-RCNN (He et al., 2017) generates bounding boxes for each instance with a powerful detector (Ren al., 2015) and categorizes each pixel in bounding boxes foreground or back- ground to obtain 28 x 28 binary grid masks. Several methods that build on Mask-RCNN improve the quality of masks. Mask Scoring RCNN (Huang et al., 2019) learns regress mask IoU to select better-quality instance masks. HTC (Chen et al., 2019) utilizes interleaved execution, mask informa- tion flow, and semantic feature fusion to improve
BMask RCNN (Cheng et al., 2020c) adds a boundary branch on Mask-RCNN to detect the boundaries of masks. Bounding Shape Mask R- C-NN (Kang et al., 2020) improves performance on object detection and instance segmentation by its bounding shape mask branch. BCNet (Ke et al., 202 uses two GCN (Welling & Kipf, 2016) lay- ers to detect overlapping instances. Although these algorithms have yielded promising results, they still restricted in the low-resolution mask representation and thus do not generate high-quality
masks.
'COCO dataset with LVIS annotations
2

---

Published as a conference paper at ICLR 2023
Devidl
Regressor
10
DCT mised patches
IDCT
Fratt
Detal PatchOTT
Classifier
M
Multi-stage
Mask obtained by DCT
PatchDCT
Classifiation
The pipeline of PatchDCT. The classifier differentiates foreground, background and mixed patches. The regressor predicts the DCT vectors of mixed patches. Masks of mixed patches are obtained by patch DCT vectors. PatchDCT combines masks of all patches obtain an entire mask of instance. The entire mask of instance output by PatchDCT can be fed into another PatchDCT module for a finer mask. For the architecture of multi-stage PatchDCT: 'F" is the feature map cropped from FPN-P2. 'M' is the high-resolution mask. "p" is the PatchDCT module.
Figure 2:
Towards high-quality instance segmentation. To take full advantage of high high-resolution masks, DCT-Mask (Shen et al., 2021) learns regress 300-dimensional DCT vector compressed from 128 x 128 mask. SOLQ (Dong et al., 2021) ) is a query-hased method, which also encodes high- resolution masks into DCT vectors and predicts the vectors by queries. Both of these methods generate high-resolution masks in a one-shot manner, without any refinement. Although they have made considerable gains, there is still potential for improvement. Multi-stage refinement is another common technique for obtaining high-quality masks. PointRend (Kirillov et al., 2020) adaptively selects several locations to refine, rendering 224 224 masks from x coarse masks. RefineMask (Zhang et al., 2021) introduces semantic segmentation masks as auxiliary inputs, and generates 112 112 masks a multi-stage manner. Mask Transfiner (Ke et al., 2022) represents image regions as quadtree and corrects the errors of error-prone tree nodes generate 112 x 112 masks. PBR (Tang ct al., 2021) a post-procee pos-processing method that refines patches along the mask boundaries. Unlike these refinement methods based on the binary grid mask representation, our method is based on compressed vectors.
Generating high-qualin masks is also of the main concerns in the field of semantic segmen- tation. CRFasRNN (Zheng et al., 2015) connects CRF (Krähenbühl & Koltun, 2011) with FCN (Long al., 2015), formulating mean-field approximate inference for the CRF with Gaussian pair- wise potentials as Recurrent Neural Networks. Deeplab (Chen et al., 2017) effectively improves the quality of masks by using atrous convolution for receptive field enhancement, ASPP for multi- scale segmentation, and CRF for boundary refinement. SegModel (Shen et al., 2017) utilizes guidance CRF to improve the segmentation quality. CascadePSP (Cheng et al., 2020b) trains inde- pendently refinement module designed in cascade fashion. RGR (Dias & Medeiros, 2018) is post-processing module based on region growing. In contrast, PatchDCT can obtain high-quality segmentation results in end-to-end learning manner without any additional post-processing.
METHODS
In this section, we show the difficulties in refining DCT vectors and then introduce PatchDCT overcome these difficulties and generate finer masks.
3.1 DIFFICULTIES IN REFINING DCT VECTORS
Given a K x K mask, DCT-Mask (Shen et al., 2021) encodes the mask M into the frequency
domain Mf. Kxk:
K-IK-1
2
(2x + 1)ur
(2y 1)UT
Mr. M u) C(w)C(v) Σ
Σ
(x,
cos
(1)
K
2K
2K
3

---

Published as a conference paper at ICLR 2023
where C(w) 1/v2 for 0 and C(w) otherwise. Non-zero values are concentrated in the
upper left corner of M K. which are low-frequency elements that contain the most information of the mask. The N-dimensiinal DCT vector is obtained by zigzag scanning (Al-Ani & Awad, 2013) M Kxk and selecting the top-N elements.
In the inference stage, M is recovered by filling the remaining elements to zero. Then each pixel in the mask Mæ* K is calculated as follow:
MKxK(Z, C(x)(())
2 K
K-1K-1
ΣΣ 4=0
M cos
(2x + 1)ur
Cos
(2y
1)eT
(2)
2K
2K
Equation reveals that each pixel in the mask MX x is calculated by all elements of M KxK- When refining the N N-dimensional DCT vector, once element incorrectly changed, all pixels in Mą* will be affected, even those correctly segmented regions, which is also shown Figure 1. Therefore, when fixing some specific error regions (e.g. borders), it is difficult get the correct refinement result unless all the elements in the DCT vector are correctly refined. In practice however, it is almost impossible correctly predict all N elements.
3.2 PATCHDCT
To prevent the above issue when refining the global DCT vector, we propose a method named patches and refines each patch respectively. The overall architecture of PatchDCT shown in Figure 2, which mainly consists of a three-class classifier and DCT vector regressor. Specifically, the classifier is used to identify mixed patches and refine foreground and background patches. Each mixed patch is then refined by an n-dimensional DCT vector, which is obtained from the DCT vector regressor.
PatchDCT, which divides the K K mask into
Three-class classifier. We define the patches with only foreground pixels and only background pixels as foreground patches and background patches, re- spectively, while the others are mixed patches. The task of differentiating patch categories is accom- plished by fully convolutional three-class classi- fier. Moreover, the mispredicted initial foreground and background patches are corrected by the classi- fier. We utilize three-class classifier instead of a DCT vector regressor to refine foreground and back- ground patches because of the particular form of their DCT vectors. For background patches, sim- ply from Equation 1, all elements of DCT vectors are zero. For foreground patches, all elements are zero except for the first element named DC com- ponent (DCC), which is equal to the patch size m. The mathematical proof of the DCT vector form for the foreground patches is shown in the Appendix. DCT vector elements of foreground and background patches are discrete data that are more suitable for classification. Referring to Figure 3, DCT vector elements of mixed patches are continuously distributed and therefore more suitable for regression.
Table 1: Mask AP obtained by different lengths of ground-truth DCT vectors using Mask-RCH RCNN framework on COCO val2017. The Ix patch size represents the binary grid mask representation. Low-dimensional DCT vectors are able to provide enough ground truth information.
Resolution
Patch Size
Dim.
AP
1I2 x 112
I 1
57.6
112 112
8 x
3
55.8
112 x 112
8 8
6
57.1
112 X 112
8
9
57.5
112 112
8 x 8
12
57.5
112 x 112
112
x 112
200
55,8
112 x 112
112
112
300
56.4
Regressor. Similar to the phenomenon described in DCT-Mask (Shen et al., 2021), refining high- resolution masks with the binary grid mask representation introduces performance degradation due to the high training complexity (refer to DCT-Mask (Shen et al., 2021) for more details). Learning to regress informative DCT vectors cases the training process. The specific experimental results are discussed in the experiments section (Sec. 4).
The regressor trained and inferred for mixed patches only. actually a boundary attention module, since the mixed patches distributed exactly along the boundary of the instance mask. For each mixed patch, the regressor predicts an n-dimensional DCT vector, which very short but highly informative. Table shows mask AP obtained by different lengths of ground truth patch DCT
4

---

Published as a conference paper at ICLR 2023
mumber eelereen
91.2
110
1.0
0.8
0.08
6.66
0.4
0.4
0.2
0.0
(a) elements of fg patches
(b) elements of bg patches
(c) elements of mixed patches
Figure 3: Elements of 6-dimensional DCT vectors for foreground, background and mixed patches on COCO va12017. DCT vector elements for foreground and background patches are discrete data. DCT vector elements for mixed patches are continuous data.
vectors using Mask-RCNN framework on COCO va/2017. The low-dimensional DCT vectors have been able provide sufficient ground truth information.
3.3 MULTI-STAGE REFINEMENT AND Loss FUNCTION
PatchDCT is module where the input and output masks have the same resolution. Thus, the mask generated by PatchDCT module can fed into another PatchDCT module for further refinement, as shown in the upper right corner of Figure 2.
With multi-stage refinement, the loss function of the mask branch is defined as
Lmask doodotxx + Σ CC Ldct..)
Ap and X2 are the loss weights. The first item Lacty of Equation 3 is the loss in predicting N- dimensional vectors the entire masks (Shen et al., 2021).
Lasty = N
1
Σ R(V. V2),,
(4)
where Vi and Vi are the i-th element in ground-trute truth and the prediction vector respectively. the loss function and N is the length of the vectors. The classification loss Lelepetch of s-th stage is the cross-entropy loss over three classes. The regression loss Låct. of s-th stage
R(V. v.) ) 7I
Σ
N
where Nm. Nan are the number of mixed patches and all patches respectively, n the length patch DCT vectors. If the k-th patch is mixed patch, pd otherwise p indicating that only DCT vectors of mixed patches are regressed.
EXPERIMENTS
4.1 DATASETS
We evaluate our method two standard instance segmentation datasets: COCO (Lin et al., 2014) and Cityscapes (Cordts al., 2016). COCO provides categories with instance-level annotations. Cityscapes is dataset focused urban street scenes. It contains categories for instance seg- mentation, providing 2.975, 500 and 1,525 high-resolution images (1, 020 2, 048) for training, validation, and test respectively.
We report the standard mask AP metric and the Boundary AP (Cheng et al., 2021) metric (AP&), the latter focusing evaluating the boundary quality. Following (Kirillov al., 2020), also report AP and AP which evaluate COCO val2017 with high-quality annotations provided by LVIS (Gupta et al., 2019). Note that for AP and AP models still trained on COCO train2017.

---

Published as a conference paper at ICLR 2023
Table 2: Mask AP on COCO with different backbones based on Mask-RCNN framework. AP" is results obtained from COCO with LVIS annotations. APB is Boundary AP. AP; is Boundary AP using LVIS annotations. Models with R101-FPN and RX101-FPN are trained with '3x' schedule. Runtime is measured on single A100. Considering the significant improvement of masks, the cost the runtime almost negligible.
Backbone
Model
AP
APs
AlM
AP
AP
AI
AI
AP.
M
AP;
AIP:
FTS
Mask-RCNN
35.2
17.2
37.7
50.3
21.1
37.6
21.3
43.7
55.1
24.8
13.9
R50-FPN
DCT-Mask
36.5
17.7
38.6
51.9
23.6
39.7
23.5
46.5
58.5
28.4
13.2
PatchDCT
37.2
18.3
39.5
54,2
24,5
40.8
23.0
47.7
60.7
30.1
12.3
Mask-RCNN
38.6
19.5
41.3
55.3
24.5
41.4
24.5
47.9
61.0
29.0
13,8
R101-FPN
DCT-Mask
39.9
20.2
42.6
57.3
26.8
43.7
25.8
50.5
64.6
32.4
13,0
PatchDCT
40.5
20.8
43.3
57.7
27.6
44,4
27.0
51.5
65.3
33.8
11.8
Mask-RCNN
39.5
20.7
42.0
56.5
25.3
42.1
25.4
48.0
61.4
29.7
13.3
RXI01-FPN
DCT-Mask
41.2
21.9
44.2
57.7
28.0
45.2
27.4
52.6
64.2
34.0
12,9
PatchDCT
41.8
22.5
44.6
58.7
28.6
46.1
27.8
53.0
66.1
35.4
11.7
Table 3: Results Cityscapes val set. APB is Boundary AP. All models based R50-FPN
backbone. PatchDCT achieves the best performance.
Methods
Resolution
AP
Al
APP
Mask-RCNN (He al., 2017)
28 x 28
33.7
60.9
11.8
Smnoptic-Deepleee (Cheng al., 2020a)
35.3
57.9
16.5
PointRender (Kirillov et al., 2020)
224 x 224
35.9
61.8
16.7
DCT-Mask (Shen al., 2021)
112 112
36.9
62.9
14.6
RefineMask (Zhang et al., 202
112 x 112
37.6
63.3
17.4
Mask Transfiner (Ke et al., 2022)
112 x 112
37.9
64.1
18.0
PatchDCT (Ours)
112 112
38.2
64.5
18.8
4.2 IMPLEMENT DETAILS
We build the model based on DCT-Mask (Shen et al., 2021). We first decode the 300-dimensional DCT vector to obtain 112 112 mask. This mask is then fed into PatchDCT together with 42 x 42 feature map cropped from FPN-P2 (Lin et al., 2017). PatchDCT refines each patch of the mask and outputs 1 112 112 mask. We set the patch size 8 and each patch is represented by a 6-dimensional DCT vector. Our model is class-specific by default, i.e. one mask per class. loss and cross-entropy loss are used for DCT vector regression and patch classification respectively. By default, only one PatchDCT module is used, and both Ag and λ₁ are set to 1. We implement our algorithm based on Detectron2 (Wu al., 2019), and all hyperparameters remain the same as Mask- in Detectron2. Unless otherwise stated, 1x learning schedule is used.
4.3 MAIN RESULTS
Results on COCO. We compare PatchDCT with Mask-RCNN and DCT-Mask over different back- bones. As shown in Table 2, on COCO val2017 with R50-FPN, PatchDCT improves 2.0% AP and 3.4% AP over Mask-RCNN, Compared with DCT-Mask, PatchDCT also achieves 0.7% AP and 0.9% AP improvements. When evaluating with LVIS annotations, PatchDCT yields significant gains of 3.2% AP" and 5.3% AP,, over MCNK-RCNN, and 1.1% AP* and 1.7% AP over DCT- Mask. Consistent improvements are observed on R101-FF and RXI01-FPN. Since AP" and AP are evaluated with high-quality annotations, the significant improvements of these two metrics em- phasize the superiority of our model. In addition, considering the improvement mask quality, the cost in runtime is almost negligible, i.c. about 1.5 FPS degradation the A.00 GPU.
We also compare the performance of PatchDCT with state-of-the-art methods of instance segmen- tation on COCO test-dev2017. With RX101 backbone, PatchDCT surpasses PointRender (Kirillov et al., 2020) and RefineMask (Zhang et al., 2021), which are both multi-stage refinement methods based on binary grid masks, by 0.8% and 0.4%. PatchDCT also achieves comparable performance with Mask Transfiner (Ke et al., 2022) with R101 backbone. However, Thanssrr runs at 5.5 FPS the A100 GPU, which is almost two times slower than PatchDCT. With Swin-B back-

---

Published as a conference paper at ICLR 2023
Table 4: Comparison of different methods on COCO test-dev2017. MS denotes using multi-scale
training. '3x" schedules indicates 36 epochs for training. Runtime is measured on single A A100.
Method
Backbone
Sched.
50
AP
AP3
AP M
AP; c
HPS
BMask RCNN (Cheng al. 2020c)
RIOI-PPN
1x
37.7
59.3
40.6
16.8
39.9
54.6
Mask-RCNN (He et al., 2017)
RI01-FPN
3x
38.8
60.9
41.9
21.8
41.4
50.5
13.8
BCNet (Ke ol,, 2021)
R101-PPN
3x
DCT-Mask (Shen et al,, 2021)
RI01-FPN
3x
40.1
61.2
43.6
22.7
42.7
51.8
13.0
Mask Transfiner (Kc et al., 2022)
R101-FPN
3x
40.7
23.1
42.8
53.8
5.5
SOLQ (Dong et al, 2021)
RI01-FPN
50c
40.9
22.5
43.8
10.7
MEInst (Zhang et al. 2020)
RX101-FPP
3x
36.4
60.0
38.3
21.3
38.8
45.7
HTC (Chen al., 2019)
RX101-FPN
20c
41.2
63.9
44.7
22.8
43.9
54.6
4.3
PointRend (Kirillov
2020)
RX101-FPN
3x
53.2
RefineMask (Zhang al., 2021)
RX101-FPN
3x
41.8
8.9
Mask Transfiner (Ke et al., 2022)
Swin-B
3x
45.9
69.3
50.0
28.7
48.3
59.4
3,5
PutcDDTT (Ours)
R101-FPN
3x
40.7
61.8
44.2
22.8
43.2
52.8
11.8
PatchDCT (Ours)
RX101-FPN
3x
42.2
64.0
45.8
25.0
44.5
53.9
11.7
PatchDCT (Ours)
Swin-B
3x
46.6
69.7
50.8
29.0
49.0
59.9
7.3
bone, PatchDCT outperforms Mask Transfiner (Ke et al., 2022) by 0.7% AP. is worth noting that PatchDCT is faster than most multi-stage refinement methods since only one refine process is quired. These results demonstrate the effectiveness of PatchDCT in generating high-quality masks.
Results on Cityscapes. We also report results Cityscapes val set in Table 3. In comparison with
PatchDCT obtains 4.5% AP and 7.0% AP improvements. It also outperforms DCT-Mask by 1.3% AP and 4.2% APB. Compared with other SOTA methods, PatchDCT is still competitive. PatchDCT achieves 0.8%, 1.4%, 2.1% APB gains over Mask Transfiner (Ke et al., 2022), RefineMask (Zhang et al., 2021) and PointRender (Kirillov et al., 2020) respectively. The large difference in APB highlights the ability of PatchDCT to generate masks with more detailed borders.
4.4 ABLATION EXPERIMENTS
We conduct extensive ablation experiments to further analyze PatchDCT. We adopt R50-FPN as the backbone and evaluate the performance on COCO val2017.
Simply refine DCT vectors. Simply refining the global DCT vectors does not succeed. To demon- strate that, we design model named 'Two-stage DCT', which regresses new 300-dimensional DCT vector after fusing the initial mask with decoded from the final DCT vector. From Table 5, Two-stage DCT achieves only little improve- ments over DCT-Mask, since changes in some elements the global DCT vector may affect the entire mask, even for the correct segmentation areas. PatchDCT leverages the patching mechanism to overcome this issue and outperforms Two-stage DCT by 1.0 AP
42 feature map from FPN-P2. The refined mask
Binary grid refinement. Refining masks with the binary grid mask representation can be considered as the extreme patching mechanism, which treats each pixel as patch. However, simply refining high-resolution masks with the binary grid mask representation introduces performance degradation. We construct an experiment named 'binary grid refinement', which predicts another 112 112 mask with the binary grid mask representation after fusing the initial mask well as a 56 x 56 feature map from FPN-P2. Experimental results in Table 5 show that the performance of binary grid refinement worse than PatchDCT, and even DCT-Mask. This is because binary grid refinement requires the refinement module to learn 12544 (112 x 112) outputs, while PatchDCT only needs to learn at most 1176 (14 x 14 x 6) outputs, which reduces the training complexity.
Effectiveness of three-class classifier. In addition to identifying mixed patches, more important role of the three-class classifier is to correct previously mispredicted foreground and background patches. To validate the effecttivneess of refining non-mixed patches (i.e. foreground and background patches), we construct a binary-class classifier, which only classifies patches as mixed or non non- mixed and keeps masks of non-mixed patches unchanged. As shown in Table 6, the binary-class classifier is inferior to our three-class classifier by 0.3% AP and 0.4% AP", since the refinement of previously incorrectly predicted foreground and background patches is ignored.
Refinement of foreground and background patches can also be accomplished with the DCT vector regressor. However, as discussed in Sec. 3.2, the DCT vector elements of the non-mixed patches
7

---

Published as a conference paper at ICLR 2023
Table 5: Mask AP obtained by different refine- ment methods on va/20/7. PatchDCT signifi cantly improves the quality of masks.
Table 6: Mask AP obtained by PatchDCT with two-class classifier and three-class classifier on val2017. PatchDCT with three-class classifier achieves the best performance.
Method
AP
AP
AP
AP
Binary grid
35.7
23.2
29.1
Clasifier 2-class
AP
AP,
PatcDDCT
Two DCT
36.6
23.9
40.1
30.1
29.1
3-class
36.9
18.2
39.3
53.5
24.4
40.4
29.7
Table 7: Mask AP obtained by PatchDCT with regressor focusing on all patches and mixed patches on val2017. The best results are ob- tained by regressing only the mixed patches.
Table 8: Mask AP obtained by PatchDCT with and without the regressor on va/20/7.
PatchDCT benefits from the regressor.
Regressor
AP
AР
AP,
Regressor
AP
APs
AP
AP,
AP,
AP"
APP
36.7
18.3
39.0
53.1
23.3
39.5
27.1
17.7
39.5
37.2
18.3
39.5
34.2
24.5
40.8
30.1
mixed
37,2
18.3
39.5
34.2
24.5
40.8
30.1
Table 9: Mask AP obtained by models with different dimensions of patch DCT vectors on COCO va/2017. Model with 6-dimensional vectors achieves the best performance.
Mask AP obtained by multi-stage PatchDCT on val2017. Two-stage PatchDCT achieves trade-off between accuracy and com- putational complexity.
Table 10:
Putch Dim.
AP
APs
AP4
AP,
AP
AP
Stage
AP
AP.
AP3
AP.
APs
AP
(GPLLPss 5.1
FPS
36.8
17.6
39.
53.5
24.0
40.3
29.3
37.2
18.3
39.5
54.1
24.5
40.8
30,1
40.0
24.7
9.6
36.9
17.1
39.3
30.1
37.3
17.3
39.7
54.6
24.7
40.9
14.1
84
Table 11:
Mask AP obtained by models with different patch sizes on COCO va/2017. PatchDCT with 8 8 patch size obtains the best performance.
Table 12: Mask AP obtained by models with different feature map sizes on COCO va/20/7. The performance saturates with the 42 42 fea- ture map.
Putch Size
AP
AP3
AP
AP,
AP
AP
Feature Size
AP
AP3
AP
AP.
AP
AP
AP
4x4
37.0
17.5
393
53.8
24.4
29.8
28
28
37.1
17.8
39.3
53.4
24.5
40.6
30.0
8x8
37.2
18.3
39.5
54.1
24.5
40.8
30.1
42 12
37,2
18.3
39.5
54.1
24.5
40.8
30.1
16x 16
37.0
17.6
39.3
53.5
24.4
56 56
37.0
17.4
39.2
53.0
24.4
41.0
30.3
Table 13: Mask AP obtained by PatchDCT with the feature map cropped from all levels and P2 only on COCO va12017. Model with the feature map of P2 obtains higher mAP.
Feature P2
AP
APs
APM
AP,
AP
AP"
AP
37.2
18.3
39.5
54.1
24.5
40.8
30.1
P2-P5
37.1
18.2
39.3
53.3
24.4
40.6
29.8
only involve zero and m, making ineffective to learn the DCT vectors of all patches directly. As shown in Table 7, the performance of the method refining non-mixed regions with the DCT vector regressor is lower than the method using three-class classifier by 0.6% AP and 1.2% AP" Need to note that, AP B and AP decrease by 0.9% and 1.5% respectively, reflecting that learning to regress non-mixed patches also affects the prediction of boundaries.
Effectiveness of the regressor. The regressor is actually a boundary attention module that generates finer boundaries. As shown in Table 8, after removing the regressor and keeping only the classifier, the overall AP only decreases by 0.5%. but APg and AP decrease by 1.2% and 3.0% respectively. The phenomenon demonstrates the importance of the regressor for generating finer boundaries.
Dimension of PatchDCT vectors We look for an appropriate patch DCT vector length to encode each mixed patch. Results Table show that the model with 6-dimensional patch DCT vectors obtains the best performance. As also shown in Table 1, the 6-dimensional patch DCT vector already contains most the ground truth information. As more elements bring only very little incremental information, regressing these elements does not improve the prediction.
Multi-stage PatchDCT, We compare the performance of the multi-stage procedure in Table 10. One-stage PatchDCT already provides high-quality masks, while two-stage PatchDCT further im- proves the prediction. However, the computational cost of the mask branch has nearly doubled with tiny improvements in the quality of masks, we choose one-stage PatchDCT in paper.
8

---

Published as a conference paper at ICLR 2023
(a) Mask-RCNN
(b) DCT-Mask
(c) PatchDCT
Figure 4: COCO example tuples from Mask-RCNN, DCT-Mask, and PatchDCT. Mask-RCNN, DCT-Mask and PatchDCT are trained based on R50-FPN, PatchDCT provides masks with higher quality and finer boundaries.
Size of the patch. We evaluate the influence of patch size in Table 11. We keep the resolution of the mask and the size of the input feature map unchanged and compare the model performance with different patch sizes. PatchDCT with 8 patches performs better than other settings.
Size of the feature map. We compare the model with different sizes of the feature map used in PatchDCT, Table 12 illustrates that the performance saturates with the 42 42 feature map.
Feature map from FPN. We evaluate PatchDCT with the feature map cropped from all pyramid levels or P2. Table 13 shows that PatchDCT benefits from the finer feature map of P2.
4.5 QUALITATIVE RESULTS
In Figure 4 we visualize some outputs of PatchDCT on COCO val2017. PatchDCT generates finer boundaries among different instances, such as the shoulder of the person (the first column), contour of the kite (the third column), and the arm of the girl (the fourth column). PatchDCT obtains masks of higher quality in comparison with Mask-RCNV and DCT-Mask.
CONCLUSIONS
In this work, we propose PatchDCT, compressed vector based method towards high-quality in- stance segmentation. In contrast previous methods, PatchDCT refines each patch of masks re- spectively and utilizes patch DCT vectors to compress boundaries that are full of details. By us- ing classifier to refine foreground and background patches, and predicting an informative low- dimensional DCT vector for each mixed patch, PatchDCT generates a high-resolution mask with fine boundaries. PatchDCT is designed with simple and clean structure, which allows the method to obtain high-quality segmentation with almost negligible cost in speed compared to Mask-RCNN and DCT-Mask. We hope that our approach will benefit future studies in instance segmentation.

---

Published as a conference paper at ICLR 2023
REFERENCES
Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):00-93, 1974.
Muzhir Shaban Al-Ani and Fouad Hammadi Awad. The jpeg image compression algorithm. Inter- national Journal of Advances in Engineering & Technology, 6(3):1055, 2013.
Kai Chen, Jiangmiao Pang. Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Zi- wei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
4974 4983, 2019.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834 848, 2017.
Bowen Cheng, Maxwell Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panop- tic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22475-12485, 2020a.
Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15334-15342, 2021.
Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward class- agnostic and very high-resolution segmentation via global and local refinement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1890-8899, 2020b.
Tianheng Cheng, Xinggang Wang, Lichao Huang, and Wenyu Liu. Moundary-preservive mask cnn. In European conference on computer vision, pp. 660-676. Springer, 2020c.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 0213-3223, 2016.
Philipe Ambrozio Dias and Henry Medeiros. Semantic segmentation refinement by monte carlo region growing of high confidence detections. In Asian Conference on Computer Vision, pp- 131-146. Springer, 2018
Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects by learning queries. Advances in Neural Information Processing Systems, 34:21898 21909, 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-enn. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6409-6418 2019.
Ba Rom Kang, Hyunku Lee, Keunju Park, Hyunsurk Ryu, and Ha Young Kim. Bshapenet: Object detection and instance segmentation with bounding shape masks. Pattern Recognition Letters, 131:449-455 2020.
Lei Ke, Yu-Wing Wing Tai, and Chi-Keung Tang. Deep occlusion- instance segmentation with overlapping bilayers. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4019-4028, 2021.
10

---

Published as a conference paper at ICLR 2023
Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi- Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4412 4421, 2022.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- lion, pp. 9799-9808, 2020.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. Fully convolutional instance-awar semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2359-2367, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3431-3440, 2015.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Falong Shen, Rui Gan, Shuicheng Yan, and Gang Zeng. Semantic segmentation via structured patch prediction, context crf and guidance crf. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1953-1961, 2017.
Xing Shen, Jirui Yang. Chunbo Wei, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, Xiaoliang Cheng, and Kewei Liang. Det-mask: Discrete cosine transform mask representation for instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Chufeng Tang. Hang Chen, Xiao Li, Jianmin Li, Zhaoxiang Zhang, and Xiaolin Hu. Look closer to segment better: Boundary patch refinement for instance segmentation. arXiv preprint arXiv:2 004.02299 2021.
Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional net- works. In J. International Conference on Learning Representations (ICLR 20/7), 2016.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https: gighhu......../. wot.://acccoooreeeeeceeettteeeee
Xue Yang and Junchi Yan. On the arbitrary-oriented object detection: Classification based ap- proaches revisited. International Journal of Computer Vision, 13005):1340-1366 2022.
Xue Yang, Jirui Yang. Junchi Yan, Yue Zhang, Tengfei Zhang, Zhi Guo, Xian Sun, and Kun Fu. Scrdet: Towards more robust detection for small, cluttered and rotated objects. In Proceedings of the IEEE International Conference on Computer Vision, pp.
Xue Yang, Junchi Yan, Ziming Feng, and Tao He. R3det: Refined single-stage detector with feature refinement for rotating object. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3163-3171. 2021a.
Xue Yang, Junchi Yan, Qi Ming, Wentao Wang, Xiaopeng Zhang, and Qi Tian. Rethinking rotated object detection with gaussian wasserstein distance loss. In International Conference on Machine Learning, pp. 1880-118411 PMLR, 2021b.
11

---

Published as a conference paper at ICLR 2023
Xue Yang, Xiaojiang Yang, Jirui Yang, Qi Ming. Wentao Wang, Qi Tian, and Junchi Yan. Learning figh-precision bounding box for rotated object detection via kullback-leibler divergence. Ad- vances in Neural Information Processing Systems, 34:18381-18394 2021c
Xue Yang, Gefan Zhang, Xiaojiang Yang, Yue Zhou, Wentao Wang, Jin Tang, Tao He, and Junchi Yan. Detecting rotated objects gaussian distributions and its 3-d generalization. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2022.
Xue Yang, Junchi Yan, Wenlong Liao, Xiaokang Yang. Jin Tang, and Tao He. Scrdet++: Detecting small, cluttered and rotated objects via instance-level feature denoising and rotation loss smooth- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2384-2333
2023.
Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, and Xiaolin Hu. Re- finemask: Towards high-quality instance segmentation with fine-grained features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6861-6869, 2021.
Rufeng Zhang, Zhi Tian, Chunhua Shen, Mingyu You, and Youliang Yan. Mask encoding for single shot instance segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10226-10235, 2020.
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Pareder Vibhav Vineet, Zhizhong Su, Da- long Du, Chang Huang, and Philip HS Torr. Conditional random fields recurrent neural net- works. In Proceedings of the IEEE international conference on computer vision, pp. 1529-1537, 2015.
Yue Zhou, Xue Yang. Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, Wenwei Zhang, and Kai Chen. Mmrotate: A rotated object de- tection benchmark using pytorch. In Proceedings of the 30th ACM International Conference Multimedia, pp. 7331-7334, 2022.
A MORE QUALITATIVE RESULTS
A.1 TWO-STAGE DCT
We visualize some outputs of two-stage DCT and compare them with DCT-Mask to demonstrate the disadvantages of simply combining DCT-Mask with multi-stage progress.
As shown in Figure 5, in two-stage DCT, the areas that were previously correctly predicted may be influenced in refinement. The phenomenon further proves the difficulties in refining DCT vectors directly.
A.2 QUALITATIVE RESULTS ON CITYSCAPES
We show some qualitative results on Cityscapes Figure 6. In comparison with Mask-RCNN and DCT-Mask, PatchDCT generates finer boundaries that greatly improve the quality of masks.
B MORE TECHNICAL DETAILS
We prove that all elements except the DCCs for foreground patches are zero.
It can be derived from Equation that DCC is equal to the patch size m in the foreground patch since M. (x. v)
m m
DCC
Σ
Mmxm (x, y) m.
(6)
m.
0
Note that for m x m patch Mf mx on (w, u) Equation can be written as
MS Mfxmm (u, = m
2
(())C(v)
m-1
Σ
A(x, u)
mm-1
Σ y=0
A(y.u)
(7)
==0
12

---

Published as a conference paper at ICLR 2023
(8)
(b)
(c)
Figure 5: Visualization of DCT-Mask (left) and two-stage DCT (right). Areas that correctly predicted are influenced by the refinement.
0%
0%
00%
%00%%
1900%
(a) Mask-RCNN
(b) DCT-Mask
(c) PatchDCT
Figure 6: Cityscapes example tuples from MAsk- DCT-Mask, and PatchDCT. Mask-RCNN. DCT-Mask and PatchDCT are trained based on R50-FPN, PatchDCT generates masks with finer boundaries.
where A(a, b) cos If is odd,
1)
2m
(2(m - x) + 1)ur
A(m 1 x, u) cos
2m
(2x 1)uT
cos
+ uT
2m
A u),
is even and larger than zero, since from Euler's formula
i8 cost isinf.
13

---

Published as a conference paper at ICLR 2023
We have
m
(2x +
1) DuT
A u)
Σ
Co8
2m
m-
Re
Σ 2=0
e
(2n+ 1jues
Re
C 2m
0,
(10)
e m
Since u is even,
umi
cos(uu) + issin(uT)
(11)
We obtain
m A(x,u) u) =0,
Vu #0,
(12)
Therefore for foreground patches
Mf xx (1.1)= m. 1= 0, otherwise.
(13)
This illustrates except the DCCs, elements of DCT vectors of foreground patches are all zero.
C LIMITATIONS AND FUTURE OUTLOOK
In the process of visualization, we observe that the model may generate masks with holes. These problems usually occur in semantical ambiguous areas, and rarely in the center of the mask where the semantic information very clear. We demonstrate some typical bad cases in Figure 7. In these cases, the model either misclassifies these patches or generates imprecise patch DCT vectors, resulting disconnected masks. We leave better classification and regression vectors future work. In addition, we also plan to carry out further verification in other more challenging areas, such as aerial images, medical images, etc. Taking aerial images as an example, this field still focuses on the research of object detection (Yang et al., 2019; 2021a;b;e; 2023), especially oriented object detection (Yang & Yan, 2022; Zhou et al., 2022; Yang et al., 2022), which lacks the exploration of more precise positioning tasks, i.e instance segmentation.
14

---

Published as a conference paper at ICLR 2023
ORing table
arning cabley
char 61%
person
TEANIS
tennis FORLN
DISTER 02%
sOn
parson
3450
DOWl
Figure 7: Visualization typical bad cases our model, PCt (left) and ground truth (right).
15


Reviews:
[{"review_id": "kXXd4iBlR4", "reviewer": "Reviewer_K4c6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is focused on instance segmentation. Baseline method applise Discrete Cosine Transform to improve the segmentation quality around the object boundary. This paper extends DCT-Mask from image-level to patch-level. Through Figure 4 we could see the quality improvement visually.", "summary_of_the_review": "The main concern is the absence of the comparision to the previous work, therefore I mark it as bordline. I am happy to change my rating if the authors can address this  question.", "strength_and_weaknesses": "The paper introduces PatchDCT, which improves the quality of instance segmentation. The experiments show the competitive performance of the proposed method. The paper provides detailed information for reproduction.\nThere some previous works also focus on the segmentation boundary, such as Fully Connected CRF in DeepLab[1], CRFasRNN[2]. Comparison to these methods maybe helpful.\n\n[1] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs\n\n[2] Conditional Random Fields as Recurrent Neural Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cMN0zkejVId", "reviewer": "Reviewer_feWT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a DCT-vector-based multi-stage refinement framework named PatchDCT, which contains a classifier and a regressor. PatchDCT first separates the original coarse mask into several patches. The classifier is used to distinguish mixed patches which consist of both foreground and background pixels. Then, these mixed patches are refined by the regressor with DCT-vector representation.\n", "summary_of_the_review": "The compressed vector-based refinement method is relatively novel. The experiment results of multiple metrics clearly show good performance in segmentation accuracy. Complementing more experiments to compare the efficiency with other refinement models could determine further superiority of the method.\n", "strength_and_weaknesses": "Strengths :\n\n1. The paper clearly identifies that straightforward refining the global DCT vetors is unsuitable and proposes a patch-based method to overcome this issue.\n2. Foreground patch and background patch have their special DCT vector, refiner them with a three-classes-classifier rather than a general DCT regressor is theoretically sound.\n3. Compared with many other methods, this work achieves SOTA results, and ablation studies are sufficient.\n\nWeaknesses :\n1. There is only runtime result compared with Mask-RCNN and DCT-Mask. Please complement more experiments to compare the efficiency of PatchDCT with other refinement models.\n2. In this paper, the result in Table 1 suggests that when using 1x1 patch and 1-dim DCT vector the network has the best performance (57.6 AP). But when encoding 1x1 patch (single-pixel) using DCT, the result should be the value of the pixel itself. What is the difference between this method and directly refining the mask with 1x1 conv when the patch size is 1x1? I think this result is inconsistent with DCT-Mask, nor \"binary grid refinement\". According to DCT-Mask (Table 1), directly increasing the resolution decreases the mask AP, which is the main reason they use DCT encoding. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IFAzJtJo6v", "reviewer": "Reviewer_NpMa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes PatchDCT for high-quality instance segmentation. Different from DCT-Mask, the whole image mask is divided into different patches. Each patch is refined individually by the classifier and regressor. The refinement is performed in a multi-stage. Improvements on mask quality are observed on COCO, Cityscapes and LVIS.", "summary_of_the_review": "The effective improvement validated by experiments; interesting idea of dividing patches into 3 classes and refining each one with classifier and regressor; clear motivation.", "strength_and_weaknesses": "Strength:\n1. Figure 1 shows clear motivation of the method. Although inspired by previous methods (such as PointRend and Mask Transfiner), dividing whole image to three classes of patches and refine the mixed patch in a multi-stage is a good strategy.\n2. Extensive ablation experiments comparison, and state-of-the-art method results (although improvement is limited).\n3. Results improvement comparing to DCT-mask with also small speed decrease.\n4. Good paper writing and clear structure, which is easy for readers to understand.\n\nWeakness:\n1.  Can the paper describes more on the speed advantages compared to previous SOTA methods? What's the speed of using one-stage PatchDCT and two-stage PatchDCT respectively?\n2. What are the typical failure/bad cases of the proposed methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AgwsSVA9eg0", "reviewer": "Reviewer_FMns", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper adds a patching technique to the DCT-Mask model and adopts a refinement technique for each patch, so that high-resolution masks can be achieved. The patching technique can produce better boundaries compared to the DCT mask model itself as element changes for DCT vectors can be limited to the patch level rather than the entire mask. The paper also provides many experiments to verify the capability of the proposed method and compare it with state-of-the-art methods. Ablation experiments are also provided to discuss the efficacy of the designed framework.", "summary_of_the_review": "In sum, the reviewer believes that the paper provides a new refinement method to improve the results of the DCT-mask model, and experiments are done to support the design of its framework. However, it is recommended that the author address the clarity issue mentioned above, and the following comments:\n1. In table 4, with the same backbone R101-FPN, SOLQ seems to have a better performance than PatchDCT with R101-FPN. However, the authors did not give any clarification in the paper.\n2. For the classifier given in the paper in Figure 2, if a foreground patch is assigned to the background, will this patch make a rectangle hole in the object?\n3. If $N$ and $n$ are the same notation, please make them consistent.\n\n---------------------------------------------------------------------------------------------------------\nUpdated: 11/27\nI appreciate that the authors well-addressed my comments and questions. I raise the score to 8.", "strength_and_weaknesses": "Strengths:\n1. Introducing the patching technique to refine the generated masks to improve boundary segmentation performance and this idea is of good interest.\n2. As patching is the key technique in the paper, the patching size is also analyzed by experiments to provide a suggested 8*8 size for users. Also, other hyperparameters like the dimension of PatchDCT vectors and the number of stages for PatchDCT are clearly given and discussed.\n3. Many experiments are done to support the design.\n\nWeaknesses:\n1. Backbone is limited to CNN-based models. Vision transformer-based model which also uses a patching technique would be of interest to see whether they will make any difference to the conclusions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster