Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes PatchDCT for high-quality instance segmentation. Different from DCT-Mask, the whole image mask is divided into different patches. Each patch is refined individually by the classifier and regressor. The refinement is performed in a multi-stage. Improvements on mask quality are observed on COCO, Cityscapes and LVIS.
strength_and_weaknesses: Strength:
1. Figure 1 shows clear motivation of the method. Although inspired by previous methods (such as PointRend and Mask Transfiner), dividing whole image to three classes of patches and refine the mixed patch in a multi-stage is a good strategy.
2. Extensive ablation experiments comparison, and state-of-the-art method results (although improvement is limited).
3. Results improvement comparing to DCT-mask with also small speed decrease.
4. Good paper writing and clear structure, which is easy for readers to understand.

Weakness:
1.  Can the paper describes more on the speed advantages compared to previous SOTA methods? What's the speed of using one-stage PatchDCT and two-stage PatchDCT respectively?
2. What are the typical failure/bad cases of the proposed methods?
clarity,_quality,_novelty_and_reproducibility: Clear writing and good paper quality; extensive experiments with effective improvement.
summary_of_the_review: The effective improvement validated by experiments; interesting idea of dividing patches into 3 classes and refining each one with classifier and regressor; clear motivation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a DCT-vector-based multi-stage refinement framework named PatchDCT, which contains a classifier and a regressor. PatchDCT first separates the original coarse mask into several patches. The classifier is used to distinguish mixed patches which consist of both foreground and background pixels. Then, these mixed patches are refined by the regressor with DCT-vector representation.

strength_and_weaknesses: Strengths :

1. The paper clearly identifies that straightforward refining the global DCT vetors is unsuitable and proposes a patch-based method to overcome this issue.
2. Foreground patch and background patch have their special DCT vector, refiner them with a three-classes-classifier rather than a general DCT regressor is theoretically sound.
3. Compared with many other methods, this work achieves SOTA results, and ablation studies are sufficient.

Weaknesses :
1. There is only runtime result compared with Mask-RCNN and DCT-Mask. Please complement more experiments to compare the efficiency of PatchDCT with other refinement models.
2. In this paper, the result in Table 1 suggests that when using 1x1 patch and 1-dim DCT vector the network has the best performance (57.6 AP). But when encoding 1x1 patch (single-pixel) using DCT, the result should be the value of the pixel itself. What is the difference between this method and directly refining the mask with 1x1 conv when the patch size is 1x1? I think this result is inconsistent with DCT-Mask, nor "binary grid refinement". According to DCT-Mask (Table 1), directly increasing the resolution decreases the mask AP, which is the main reason they use DCT encoding. 
clarity,_quality,_novelty_and_reproducibility: For one thing, the work starts with the issue of straightforward refining the global DCT vectors, proposes a patch-based method to solve this issue and finally gets well performance. It's logically fluent and complete. The experiment result of multiple metrics on three popular instance-segmentation datasets is also clear.

For another, as a refinement method, the work clearly shows that very small-size compressed vectors (such as 6-dimension) can afford enough information for segmentation. This is a valuable result for the refinement task.
summary_of_the_review: The compressed vector-based refinement method is relatively novel. The experiment results of multiple metrics clearly show good performance in segmentation accuracy. Complementing more experiments to compare the efficiency with other refinement models could determine further superiority of the method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is focused on instance segmentation. Baseline method applise Discrete Cosine Transform to improve the segmentation quality around the object boundary. This paper extends DCT-Mask from image-level to patch-level. Through Figure 4 we could see the quality improvement visually.
strength_and_weaknesses: The paper introduces PatchDCT, which improves the quality of instance segmentation. The experiments show the competitive performance of the proposed method. The paper provides detailed information for reproduction.
There some previous works also focus on the segmentation boundary, such as Fully Connected CRF in DeepLab[1], CRFasRNN[2]. Comparison to these methods maybe helpful.

[1] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

[2] Conditional Random Fields as Recurrent Neural Networks
clarity,_quality,_novelty_and_reproducibility: The writing is good and presentation is clear.
summary_of_the_review: The main concern is the absence of the comparision to the previous work, therefore I mark it as bordline. I am happy to change my rating if the authors can address this  question.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper adds a patching technique to the DCT-Mask model and adopts a refinement technique for each patch, so that high-resolution masks can be achieved. The patching technique can produce better boundaries compared to the DCT mask model itself as element changes for DCT vectors can be limited to the patch level rather than the entire mask. The paper also provides many experiments to verify the capability of the proposed method and compare it with state-of-the-art methods. Ablation experiments are also provided to discuss the efficacy of the designed framework.
strength_and_weaknesses: Strengths:
1. Introducing the patching technique to refine the generated masks to improve boundary segmentation performance and this idea is of good interest.
2. As patching is the key technique in the paper, the patching size is also analyzed by experiments to provide a suggested 8*8 size for users. Also, other hyperparameters like the dimension of PatchDCT vectors and the number of stages for PatchDCT are clearly given and discussed.
3. Many experiments are done to support the design.

Weaknesses:
1. Backbone is limited to CNN-based models. Vision transformer-based model which also uses a patching technique would be of interest to see whether they will make any difference to the conclusions.
clarity,_quality,_novelty_and_reproducibility: Clarity:
In general, it is clear. However, in the 4.3 section, the second paragraph did not clearly give reference to its data points. Further, it mentioned that "MaskTransifer runs at 5.5 FPS on the A100 GPU, which is almost two times slower than PatchDCT", however, the paper did not provide concrete data points.

Quality:
In general, the paper is well-written.

Novelty:
The reviewer believes that the novelty of this paper is good.
summary_of_the_review: In sum, the reviewer believes that the paper provides a new refinement method to improve the results of the DCT-mask model, and experiments are done to support the design of its framework. However, it is recommended that the author address the clarity issue mentioned above, and the following comments:
1. In table 4, with the same backbone R101-FPN, SOLQ seems to have a better performance than PatchDCT with R101-FPN. However, the authors did not give any clarification in the paper.
2. For the classifier given in the paper in Figure 2, if a foreground patch is assigned to the background, will this patch make a rectangle hole in the object?
3. If $N$ and $n$ are the same notation, please make them consistent.

---------------------------------------------------------------------------------------------------------
Updated: 11/27
I appreciate that the authors well-addressed my comments and questions. I raise the score to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In a gist, the paper proposes mainly to use Discrete Cosine Transforms at a patch and not just image level, so that improve segmentation quality around boundaries.

This paper received strong acceptance scores from 3 out of 4 reviewers. They all acknowledge the idea is novel enough, the experiments convincingly contribute to the literature, and the writing is clear so that to be positively accepted by the community. 

There was criticism on whether the positioning and comparisons with related work is sufficient. The authors include theoretical comparison in the related work, they also show that the method is similarly efficient with Mask-RCNN and DCT-Mask in terms of FPS, while using same backbone architectures.

All in all, under scrutiny, this paper was well received, and I recommend acceptance. 
justification_for_why_not_higher_score: In terms of a 'surprise factor' that I think is important on whether a paper should be recommended as a spotlight or an oral, I think the paper is lacking. The main idea is close to what other papers have proposed (DCT for segmentation), but a clever combination and/or application improves results further. All in all, I think the paper brings a solid contribution and a poster presentation is sufficient.
justification_for_why_not_lower_score: See metareview.
Decision: Accept: poster