Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a point affiliation for feature upsampling which is one of the most essential parts, especially dense prediction networks. The proposed method generates similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features. It also introduce a lightweight upsampling operator, termed Similarity-Aware Point Affiliation (SAPA) and its variant. Experiments show the superiority of the proposed upsampling module on various depth prediction tasks.
strengths_and_weaknesses: + Introducing the notion of point affiliation into feature upsampling is interesting and makes sense. Many other followers will benefit from such notions. 
+ In depth comparison to other previous upsampling methods such as CARAFE or IndexNet, e.g., in Fig. 2, is interesting.
+ In most cases, the state-of-the-art performance is attained when it is incorporated with various dense prediction networks.

questions: - Learning feature upsampling is interesting and makes sense. But this framework requires additional learnable parameters, which may make the networks more suffer from the overfitting problem, less generalizable. It would be great if there are experiments about this overfitting issue and generalization issues.
- In Kernel generation part, the similarity between encoder feature and decoder feature is computed. But, such encoder feature and decoder feature may have different data distributions, so directly computing the similarity between them may be sub-optimal. To overcome this, very simple normalization prior to computing similarity may be used. It would be great if the relevant comments or experiments are additional conducted.
- The proposed module, SAPA, consists of many sub-modules, e.g., Y embedding, X embedding, Gated addition, Kernel generation, etc. So through experiments for ablation study is required. The current ablation study in Table 4 did not cover all the variants of ablation study. 

Minor comments:
- Using subscript of subscript, e.g., in Line 184, may make the reader follow the paper. It would be better if simpler notations are used.
- In Similarity Function in Line 203-205, why "low-rank" versions are used? Please clarify this.
limitations: The paper discussed the limitation of the proposed method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents an interesting feature unsampling module, which can be flexible applied to the tasks with upsampling like segmentaiton, detection and depth estimation. The main idea is to generate the kernels based on the feature clustering similarity. It provides extensive experiments on different dense prediction tasks and consistent performance gain has been obtained. The paper is well motivated and the presentation is clear. 
strengths_and_weaknesses: strength:
1. The idea to design a feature upsampling framework based on the clustering similarity is interesting and novel.
2. The proposed module obtains state-of-art performance on dense prediction tasks like segmentation, detection, and depth estimation, without large computational overhead.
3. The proposed module is well motivated and the presentation of the paper is clear. 

weakness:
1. As shown in Table 3, the performance of on object detection is a little bit lower than the baseline of CARAFE [1]. Although the authors provide an explaination in Section 6, usually the better segmentation results should lead to better localization on the bounding-box level. 
2. Compared with the baselines, the implementation of the proposed approach is a little bit complicated with more steps.
questions: 1. The segmentation experiments are based on three transformer-based models. How about the segmentation performance for the CNN-based approaches?
2. The proposed approach relies on the similarity score. It is similar to a new interpolation between the features. Thus, how about is result of the high-frequency data if the ground-truth mask within a  local neighborhood changes abruptly?
limitations: The paper claims the potential limitation on the object detections which requires on the semantic mending.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a new approach for upsampling decoder features with the guidance of encoder feature, leading to semantic preserving and detail delineation. The key idea is to apply similarity-aware upsampling based on encoder and decoder features. To be more specific, the decoder feature is upsampled via the weighted sum in the sub-region, where the weight is calculated using the similarity between encoder and decoder features, as shown in Figure 3. 
strengths_and_weaknesses: +) The dense prediction tasks (semantic segmenation, depth etimation, and image matting) often require an accurate and detail-preserving upsampling of decoder features, and the proposed simple upsampling scheme can be very effective with no significant computational overhead.

+) Experiments validated the performance gain on the above-mentioned tasks.

-) The overall idea is very simple, and it can be explained using (2) and (3) together with Figure 3. Nevertheless, the manuscript needs to be revised in a more compact form. In the abstract and introduction, authors stated that the point affiliation (semantic cluster) should be incorporated in the upsampling process of decoder features. 'Cluster' seems to be rather misleading. The proposed idea is just a similarity based summation, where the similarity is measured between encoder and decoder features. There is no explicit clustering and affiliation estimation process in the proposed framework.  

-) The equation (4) is rather meaningless. The proposed kernel is rather similar to joint bilateral filter, and it is well-known that in the smooth region, the joint bilateral kernel becomes the Gaussian function. Though the proposed kernel becomes a uniform function, such a derivation seems unnecessary in the main paper (maybe it can be moved to supplementary material).

**********************************************************************************
* After rebuttal

Authors addressed some questions in the rebuttal. I appreciate it.

This work introduces a simple yet effective approach for dense prediction tasks. This kind of upsampling (using two inputs) has been adopted in various vision tasks, but using it in the upsamling process for decoder features sounds interesting.
Nevertheless, considering the simplicity of the overall method, I would like to keep the initial rating (Borderline accept).
**********************************************************************************

questions: - In Figure 3, the decoder feature is added into the encoder feature. This was not explained in the paper.

- The similarity kernel w_{i1, j1, m, n}, which is used for obtaining the upsampled decoder feature at (i1, j1), is calculated with the single encoder feature at (i1, j1) and the set of decoder features (m,n) for m=-r,...,r and n=-r,...,r. This means that upsampling relies on the single encoder feature at (i1, j1), and thus it seems that the proposed method is still too sensitive to the noise of the encoder feature.

limitations: N.A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new kernel upsampling module which encourages not only semantic smoothness but also boundary sharpness in order to enhance the performance of semantic segmentation/matting tasks. The key idea is to design a similarity-aware kernel which compare the similarity between encoded features with local spatial awareness about the decoded features. The paper has also proposed a lightweight upsampling operator,  Similarity-Aware Point Affiliation (SAPA), which demonstrated high performance on various dense prediction tasks. The proposed method has compared with CARAFE, IndexNet and A2U in term of both quantitative results and computational complexity. 
strengths_and_weaknesses: Overall, this is a solid paper and I think the proposed method is very useful in dense prediction tasks that require very accurate boundary. The experimental comparisons are also sufficient to validate the performance of the proposed method, and I think its simplicity would make this method useful in real world applications.
questions: Although table 1 has summarize the theoretical complexity of various, I would like to see the actual runtime of the proposed method compared with other methods. 

Besides the quantitative results in table 2 and table 3, I would like to see some qualitative results which show the boundaries of the segmented regions, especially for the matting tasks. 

The ablation study about the kernel size seems to be too simple, I hope there is a deeper study about its effectiveness.

limitations: The paper has discussed it properly.
review_title: null

Meta-review: metareview: The paper focuses on the task of feature upsampling, specifically in decoder layers for dense prediction problems. The proposed point affiliation module can be used in upsampling kernels to produce semantically smooth and boundary preserving upsampled sets. The paper received four detailed reviewers from experts. There was a healthy discussion between authors and reviewers during the discussion period and the extra analyses, explanation, and experiments from the authors helped resolve most of the concerns raised by the reviewers. With these extra items presented in the discussion period, the paper has reached the level of impact and contribution expected by NeurIPS papers. The authors are recommended to include them in the final version of the paper.
confidence: Less certain
Decision: Accept