SAPA: Similarity-Aware Point Affiliation for Feature Upsampling
Authors: Hao Lu
Wenze Liu
Zixuan Ye
Hongtao Fu
Yuliang Liu
Zhiguo Cao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Feature upsampling
dense prediction
semantic segmentation
object detection
depth estimation
image matting

Abstract: We introduce point affiliation into feature upsampling, a notion that describes the affiliation of each upsampled point to a semantic cluster formed by local decoder feature points with semantic similarity. By rethinking point affiliation, we present a generic formulation for generating upsampling kernels. The kernels encourage not only semantic smoothness but also boundary sharpness in the upsampled feature maps. Such properties are particularly useful for some dense prediction tasks such as semantic segmentation. The key idea of our formulation is to generate similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features. In this way, the encoder feature point can function as a cue to inform the semantic cluster of upsampled feature points. To embody the formulation, we further instantiate a lightweight upsampling operator, termed Similarity-Aware Point Affiliation (SAPA), and investigate its variants. SAPA invites consistent performance improvements on a number of dense prediction tasks, including semantic segmentation, object detection, depth estimation, and image matting. Code is available at: https://github.com/poppinace/sapa

SAPA: Similarity-Aware Point Affiliation for Feature Upsampling
Hao Lu
Wenze Liu Zixuan Ye Hongtao Fu Yuliang Liu Zhiguo Cao"
School of Artificial Intelligence and Automation Huazhong University Science and Technology
Wuhan 430074, China (hlu, @hust. wwt.tthuuuuuuuu.ueeeeeeee edu. cn
Abstract
We introduce point affiliation into feature upsampling, notion that describes the affiliation of each upsampled point to a semantic cluster formed by local de- coder feature points with semantic similarity. By rethinking point affiliation, we present a generic formulation for generating upsampling kernels. The kernels encourage only semantic smoothness but also boundary sharpness in the up- sampled feature maps. Such properties are particularly useful for some dense prediction tasks such semantic segmentation. The key idea of our formula- tion is to generate similarity- kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features. In this way, the encoder feature point can function as a cue to inform the semantic cluster of upsampled feature points. To embody the formulation, we further instantiate lightweight upsampling operator, termed Similarity-Awarc Point Affiliation (SAPA), and investigate its variants. SAPA invites consistent performance improvements number of dense prediction tasks, including se- mantic segmentation, object detection, depth estimation, and image matting. Code is available at: https:/ github. ctt/ http///pppncccsaaapeeee
Introduction
We introduce the notion of point affiliation into feature upsampling. Point affiliation defines a relation between each upsampled point and a semantic cluster" which the point should belong. It highlights the spatial arrangement of upsampled points at the semantic level. Considering an example shown in Fig. 1 w.r.t. x2 upsampling in semantic segmentation, the orange point of low resolution will correspond to four upsampled points of high resolution, in which the red and yellow ones should be assigned the 'picture' cluster and the 'wall" cluster, respectively. Designating point affiliation difficult and sometimes can be erroneous, however.
In x2 upsampling, nearest neighbor (NN) interpolation directly copies four identical points from the low-res one, which assigns the same semantic cluster to the four points. On regions in need of details, the four points probably do share the same cluster but are forced to share. Bilinear interpolation assigns point affiliation with distance priors. Yet, when tackling points of different semantic clusters, it not only cannot inform clear point affiliation, but also blurs the boundary between different semantic clusters. Recent dynamic upsampling operators have similar issues. CARAFE [1. 2] judges the affiliation of an upsampled point with content-aware kernels. Certain semantic clusters will receive larger weights than the rest and therefore dominate the affiliation of upsampled points. However, the affiliation near boundaries or on regions full of details can still be ambiguous. As shown Fig. the
Corresponding author.
A semantic cluster is formed by local decoder feature points with the similar semantic meaning.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

x=x
G
Kand
Liparppte
Kerel...
High-resolution C2 Low-resolution decoder feature
C2
m//r
cucoder Rature
Upsereple
High-resolurion decoder feature
Kend
Encoder feature
Decoder feature
Upsemppeed Resture
Figure Left: Similarity between an encoder point and different semantic clusters in the decoder. Right: Point affiliation mechanism of ideal upsampling kernels. Left: If the red-box encoder feature point classified into the semantic cluster C1, then is more similar C₁ than Cz- Right: The upsampling kernels can be 'soft' selector in a local window to assign point affiliation. For an upsampled point, the kernel selects a/some representative points from its most relative semantic cluster. E.g., according to the encoder feature, the red upsampled feature point should belong to the 'picture' cluster. Then we expect the kernel can assign large weights on picture-related points and small weights on wall-related points. In this way, after the weighted sum, the upsampled point will be revalued and assigned the 'picture' cluster.
boundaries are unclear in the feature maps upsampled by CARAFE. The reason is that the kernels are conditioned on decoder features alone; the decoder features carry little useful information about high-res structure.
Inferring structure requires high-res encoder features. For instance, if the orange point in Fig. lies on the low-res boundary, is difficult to judge to which cluster the four upsampled points should belong. However, the encoder feature in Fig. 1 actually tells that, the yellow point is on the wall, and the red one is on the picture, which suggests one may extract useful information from the encoder feature to assist the designation of point affiliation. Indeed IndexNet [3, 4] and A2U [5] have attempted to encode such information to improve detail delineation in encoder-dependent upsampling kernels: however, the encoder feature can easily introduce noise into kernels, engendering discontinuous feature maps shown in Fig. 2. Hence, the key problem seems to be how to extract only required information into the upsampling kernels from the encoder feature while filtering out the rest.
To use encoder features effectively, an important assumption of this paper is that, an encoder feature point is most similar to the semantic cluster into which the point will be classified. Per the left of Fig. 1, suppose that the encoder point in the red box is assigned into the cluster C1 by its semantic meaning, then it is similar to C1, while not similar to C2. As result, by comparing the similarity between the encoder feature point and different semantic clusters in the decoder feature, the affiliation of the upsampled point can be informed according to the similarity scores. In particular, we propose to generate upsampling kernels with local mutual similarity between encoder and decoder features. For every encoder feature point, we compute the similarity score between this point and each decoder feature point in the spatially associated local window. For the green point in Fig. I, since every point in the window shares the same semantic cluster, the encoder feature point is as similar as every point in the window. In this case we expect an 'average kernel' which is the key characteristic to filter noise, and the upsampled four points would have the same semantic cluster as before. For the yellow point in the encoder, since it belongs to the 'wall' cluster, it is more similar to the points on the wall than those on the picture. In this case we expect kernel with larger weights on points related the 'wall' cluster. This can help to assign the affiliation of the yellow point to be in the 'wall' cluster.
By modeling the local mutual similarity, derive generic form of upsampling kernels and show that this form implements our expected upsampling behaviors: encouraging both semantic smoothness and boundary sharpness. Following our formulation, we further instantiate a lightweight upsampling operator, termed Similarity- Point Affiliation (SAPA), and investigate its variants. We evaluate SAPA across number of mainstream dense prediction tasks, for example: i) semantic segmentation: we test SAPA on several transformer- based segmentation baselines on the ADE20K dataset [6], such as SegFormer [7], MaskFormer [8], and Mask2Former [9], improving the baselines by 1 1% 2.7% mloU; ii) object detection: SAPA improves the performance of Faster R-CNN by 0.4% AP on MS COCO [10]; iii) monocular depth estimation: SAPA reduces the rmse metric of BTS [1]] from 0.419 to 0.408 on NYU Depth V2 [12]: and iv) image matting: SAPA outperforms strong A2U matting baseline [5] on the Adobe Composition- 1k testing set [13] with further 3.8% relative error reduction in the SAD metric. SAPA also outperforms at least is on par with other state-of-thee thtt--r

---

Upsampled maps
Kenal
Image & Label
Max unpool
Bilinear
CARAFE
IndexNet
A2U
SA
Additional kemel visualizations SAPA
Figure 2: Top: Upsampled feature maps and upsampling kernel maps of different upsampling operators. Down: Additional upsampling kernel maps of SAPA. The visualizations are produced with SegNet [14] the baseline on the SUN RGBD [15] dataset. For each upsampling operator, choose the first three channels from the feature maps of the last upsampling stage for visualization. Only SAPA shows both smooth regions and sharp boundaries. The kernel map of CARAFE is coarse and lacking in details, while IndexNet and A2U generate kernels with much undesired details from the encoder. See the supplementary material for additional visualizations.
dynamic upsampling operators. Particularly, without additional parameters, SAPA outperforms the previous best upsampling operator CARAFE on semantic segmentation.
2 Related Work
We review work related feature upsampling. Feature upsampling is fundamental procedure in encoder-decoder architectures used recover the spatial resolution of low-res decoder feature maps and has been extensively used in dense prediction tasks such as semantic segmentation [14, 7, 8, 9] and depth estimation [16, 17, 11].
Standard upsampling operators are hand-crafted. NN and bilinear interpolation measure the semantic affiliation in terms relative distances in upsampling, which follows fixed rules designate point affiliation, even if the true affiliation may be different. Max unpooling [14] stores the indices of max-pooled feature points encoder features and uses the sparse indices to guide upsampling. While it executes location-sped specific point affiliation which benefits detail recovery, most upsampled points are assigned with null affiliation due to zero filling. Pixel Shuffle [18] is widely-used in image/video super-resolution. Its upsampling only includes memory operation reshaping depth channels to space ones. The notion of point affiliation does not apply to this operator, however.
Another stream of upsampling operators implement learning-based upsampling. Among them, transposed convolution deconvolution [19] is known as an inverse convolutional operator, Based on novel interpretation of deconvolution, PixelTCL [20] is proposed to alleviate the checkerboard artifacts [21] of standard deconvolution. In addition, bilinear additive upsampling [22] attempts to combine learnable convolutional kernels with hand-crafted upsampling operators to achieve composited upsampling. Recently, DUpsample [23] seeks to reconstruct the upsampled feature map with pre-learned projection matrices, expecting to achieve data-dependent upsampling behavior.

---

kernel While these operators are learnable, the upsampling kernels are fixed once learned, still resulting in ambiguous designation of point affiliation.
In learning-basee based upsampling, some recent work introduces the idea of generating content-aware dynamic kernels. Instead of learning the parameters of the kernels, they learn how to predict the kernels. In particular, CARAFE [1, 2] predicts dynamic kernels conditioned on the decoder features. IndexNet [3, 4] and A2U [5]. by contrast, generate encoder-dependent kernels. While they significantly outperform previous upsampling operators in various tasks, they still can cause uncertain point affiliation, resulting in either unclear predictions near boundaries or fragile predictions
in regions.
Our work is closely related to dynamic kernel banee- upsampling. We also seek to predict dynamic kernels; however, we aim to address the uncertain point affiliation in prior arts to achieve simultaneous region smoothness and boundary sharpness.
3 Dynamic Upsampling Revisited
We first revisit two key components shared by existing dynamic upsampling operators: generation and feature assembly.
Kernel Generation Given the decoder feature X' #xWxC if we upsample it to the target feature X' E 2N then for any point at the location I' = J') in X', we generate a kernel Wy based on the neighborhood feature Ny that spatially corresponds to l'_ In this way, kernel generation can be defined by
Wi norm(w
where v refers to kernel generation module, which is often implemented by sub-network used to predict the kernel conditioned on Nr, and norm can originate from two sources, to be specific, from the encoder feature y E 22 x2W xC or from the decoder feature X' If the encoder feature y is chosen as the source, then a local region of y centered at (' is extracted to be Ny. If the decoder feature X' is chosen, one first needs to compute the projective location of i.e., = (i.j) = 1t'l = (1="1 1 according to the spatial location correspondence, then local region of X' centered at / is regarded as Ny. The sof tmax function is often used as the normalization function such that relevant points can be softly selected to compute the value of the target point using the weight Wy.
normalization function. The feature AG
According Fig. 2, the source of Ny affect the predicted kernel. The kernels predicted by CARAFE, IndexNet, and A2U show significantly distinct characteristics. With the decoder feature alone, the kernel map coarse and lacking in details. Benefiting from the encoder feature, the kernel maps generated by IndexNet and A2U have rich details; however, they manifest high similarity to the encoder feature, which means noise is introduced into the kernel.
Feature Assembly For each target feature point at I', we assemble the corresponding sub-region of decoder feature with the predicted K x K kernel Wy, whose weight denoted by Wr.p, E I, {(u, -r, .r, r 2 }}, to obtain the value of the target point X². by a weighted sum
X
Wr Wr.
(2)
PET
By executing feature assembly on every target feature point, we can obtain the target upsampled feature map. As shown in Fig. the upsampled feature has close relation to the kernel. A well- predicted kernel can encourage both semantic smoothness and boundary sharpness; a kemel without encoding details or with too many details encoded can introduce noise. We consider an ideal kernel should only response at the position in need of details, while do not response (appearing as an average value over an area) at good semantic regions. More importantly, an ideal kernel should assign weights reasonably so that each point can be designated to correct semantic cluster.
4 Rethinking Point Affiliation with Local Mutual Similarity
To obtain an expected upsampling kernel mentioned above, we first derive a generic formulation for generating the upsampling kernel by exploiting local mutual similarity, then explain why the formula-

---

tion encourages semantic smoothness and boundary sharpness, and finally present an upsampling operator, SAPA, that embodies our formulation.
4.1 Local Mutual Similarity
We rethink point affiliation from the view of local mutual similarity between encoder and decoder features. With detailed analysis, explain why such similarity can assist point affiliation.
We first define generic similarity function samn(a y) I& x Rd + IR. scores the similarity
between vector x and vector y of the same dimension d. We also define normalization
function involving real numbers 21, 22, En by norm(r;; 01, 01,
-... 22,, Fnn) ン h(2z)) where
h(x): R R is an arbitrary function, ignoring zero division. Given sim( y) and hzz). we can
define a generic formulation for generating the upsampling kernel
h (sim(( ax. y))
(3)
Σ h (sim(x', y))
N'ENG
where is the kernel value specific to a and y. To analyze the upsampling behavior of the kernel, we further define the following notations.
Let Br E RC denote the encoder feature vector at position l' and æ; E RC be the decoder feature vector position I, where C is the number of channels. Our operation will be done within local window of size K x K, between each encoder vector y/ and all its spatially associated decoder feature vectors, El+p's s.
To simplify our analysis, we also assume local smoothness. That is, points with the same semantic cluster will have a similar value, which means local region will share the same value on every channel of the feature map. As shown in Fig. 1, we choose a and b as the feature vectors in 'wall' and 'picture' cluster, respectively, then a and b are constant vectors. For ease of analysis, we define two types of windows distinguished by their contents. When all the points inside a window belong to the same semantic cluster, is called a smooth window; while different semantic clusters appear in window, it is defined as a detail window.
Next, explain why the kernel can filter out noise, why it encourages semantic smoothness in smooth window, and why it can help to recover details when dealing with boundaries/textures in a detail window.
Upsampling in a Smooth Window Without loss of generality, we consider an encoder point at the position I' Fig. 1. corresponding window is smooth window of the semantic cluster 'picture', thus Pttp = b. Vp E T. Then the upsampling kernel weight w.r.t. the upsampled point l' at the position p takes the form
Msim( Mr)) A(sim(b yv))
Wy = norm norm(sim(a Dy
(4)
Σ Mssim( Br)) K*h(ssimbb, bv)))
K
which has nothing to do with 1, and p. Eq. (4) reveals a key characteristic of local mutual similarity in smooth window: the kernel weight is constant regardless of y. Therefore, the kernel fundamentally can filter out noise from encoder features with an 'average' kernel.
Note that, in the derivation above the necessary conditions include: i) x is from a local window in the k(x,))
decoder feature map; ii) normalization function in the form of Σ, k(,y)
Upsampling in a Detail Window Again we consider two encoder points at the position m'' and n' in Fig. 1. Idcally yn and y should be classified into the semantic cluster of 'wall' and 'picture'. respectively. Taking ym as an example, following our assumption, it is more similar points of the 'wall' cluster rather than the 'picture' cluster. From Eq. (4), we can tell that sim (x Un is larger than sim(# Fmm where and f are the offsets in the window such that m + and m + are within the "wall' and the 'picture' cluster, respectively. Therefore, after computing similarity scores and normalization, one can acquire kernel with significantly larger weights on points with the semantic cluster of "wall' than that of 'picture', i.e., Wm. >> Wm.' After applying the kernel to the corresponding window, the upsampled point at m' will be revalued and assigned to the semantic cluster of 'wall'. Similarly, the upsampled point at n' will be assigned into 'picture'.
5

---

generation
Feature assembly
processing
HE
mann malipization
lince crboabiing yuiing modiatioes
Pepeeesing
X
X
Enoode festure
kenck
Computing similarity
Uparipted
@4 c
Deooder
Figure 3: Feature upsampling of SAPA. We compute mutual similarity scores with a similarity function between each encoder feature point and the spatially associated local decoder feature points (features are normalized with LayerNorm before similarity computation). The scores are then transformed into upsampling kernel weights after kernel normalization. The kernel is then used to assemble the upsampled feature points. We illustrate three variants: SAPA-I with inner-product similarity, SAPA-B with (low-rank) bilinear similarity, and SAPA-G with gated (low-rank) bilinear similarity. They differ in the pre- pre-procsssing step. See detailed definition in Section 4.2.
Note that, in Eq. (4) we have no constraint on y. But here in detail window, y as an encoder feature vector can play vital role for designating correct point affiliation. Next our concrete implementation, we discuss how to appropriately use in the similarity function.
4.2
SAPA: Similarity-Aware Point Affiliation
Here we embody Eq. (3) by investigating different similarity and normalization functions.
Normalization Function Though we do not constrain h(x) in theory, in reality it must be carefully chosen. For example, to avoid zero division, the scope for the choice of h(x) is narrowed. Following existing practices [1, 3, 5], we choose h(z) e" by default, which is equivalent to softmax normalization. We also test some other h(zr)ss s. such as h(xx) Re)U(x)) h(x) = si sigmoid(xr). and M(x)= = softplus(r). Their performance comparisons will be given in ablation studies.
Similarity Function We study three types of similarity functions:
Inner-product similarity: sim(æ, v) x y
(Low-rank) bilinear similarity [24]: sim(æ, y) = xx P P.y.
Gated (low-rank) bilinear similarity [25]: sim(x, y) gxx PT (1- PT PTPP P.
where Pa E RR Py E RR Par E RRdxC and Pam:: E RdxC are the linear projection matrices, d is the embedding dimension, and g (0, 1) is a gate unit learned by linear projection. We model the projection matrices to be low-rank, i.e., d < C, to reduce the number of parameters and computational complexity. The gate-modee modulated bilinear similarity is designed to further filter out the encoder noise. The gate is generated by learning a linear projection matrix that projects the decoder feature to a single-channel mask, followed by sigmoid normalization. Finally, we have ) GY + (1 ( (nearest neighbor interpolation is used for matching the resolution), where G is the matrix form the gate unit. The use of the gate implies we reduce noise in some spatial regions by replacing encoder features y' with decoder features X. Its vector form explains that in the area of noise it tends to switch to the self-similarity mode. We will prove the effectiveness of the gating mechanism by comparing the gate-modulated bilinear similarity with a baseline without the gating mechanism.
In practice, encoder and decoder features have different data distributions, which is unsuitable to compare similarity directly. Therefore LayerNorm is applied to encoder and decoder features before similarity computation. Indeed we observe that the loss does not decrease without normalization.
As shown in Fig. 3, our implementation is similar to the previous dynamic upsampling operators. which first generates the upsampling kemnels and then assembles the decoder feature conditioned on the kernels. We highlight the kernel generation module. By setting a kernel size K, for each encoder feature point, we compute the similarity scores between this point and each of K K neighbors in the decoder. Then, the tmax normalization is applied to generate the upsampling kernels. SAPA is lightweight and can even work without additional parameters (with inncr-product similarity).

---

Table 1: Computational complexity and parameters of SAPA and other upsamplers. C: number of channels, d: embedded channel number. As the default settings, = 64, K 5 in CARAFE and K in A2U. We set d 32 and K 5 in SAPA. I: inner-product similarity; B: bilinear similarity; G: gated bilinear similarity.
Module
Operation
(XW)
Params
CARAFE
Kernel generation
Cd + 36K 4K
Od + 36K d 0 Cd + 36366
Feature ussembly
Total
Cd + 36K d + 4K
IndexNet
Kerel generation
32C² 8C
32C² 8C
HIN
Feature assembly
10 32C + 12C
0
Total
320 + 80
IndexNet
Kemel generation
6SC
68C
M20
Feature assembly
Total
68C 4C
680
A2U
Kemel generation
78C 4 4K?
4 - 20
Feature assembly
AK3C
0
4K 4K
4K 20
SAPA
(1) Feature embedding
6od
20d
(2) Gated addition
C + 8d 4K'd
C
(3) Inner product
0
(4) assembly
4K3C
0
(3)(4)
SK2C
Total (T)3)(4)
SCd 4K 4
2Cd
G Total (1X2133X4)
sCd Nod+4K344 1K +
2Cd + C
To intuitively understand its lightweight property, we compare the computational complexity and number of parameters of different dynamic upsampling operators in Table 1. For example, when upsampling a 256-channel feature map, the FLOPs are H * W * 99K, H * W 2M, H * W 28K, and H W 70K for CARAFE, IndexNet-HIN, A2U, and SAPA-B, respectively.
visualize the feature maps of upsampling kernels and upsampled features Fig. Our up- sampling kernels show more favorable responses than other upsampling operators, with weights highlighted on boundaries and noise suppressed in regions, which visually supports our proposition and is concrete embodiment of Eq. (4).
Experiments
We first focus our experiments on semantic segmentation to justify the effectiveness of SAPA. We then showcase its universality across three additional dense prediction tasks, including object detection, depth estimation, and image matting. All our experiments are run on a server with 8 NVIDIA GeForce RTX 3090 GPUs.
5.1 Data Sets, Metries, Baselines, and Protocols
For semantic segmentation, conduct experiments the ADE20K dataset [6] and report the mloU metric. Three strong transformer-based models are adopted as the baselines, including SegFormer- B1 [7]. Maskkorrmer-Siin-Bae
[8] and Fansk2Frrmr--iinntt WWin-BaseeB [9], where the Swin-Base backbone pretrained on I2ageNe All training settings and implementation details are kept the same as the original papers. We only modify the upsampling stages with specific upsampling operators.
For object detection, we use the MS COCO [10] dataset, which involves S0 object categories. We use AP the evaluation metric. Faster R-CNN [26] with ResNet-50 [27] is adopted as the baseline. We use mmdetection [28] and follow the xx (12 epochs) training configurations.
For depth estimation, we use NYU Depth V2 dataset [12] and its default train/test split. We choose BTS [11] with ResNet-50 as the baseline and follow its training configurations. The standard depth metrics used by previous work is employed to evaluate the performance, including root mean squared error (RMS) and its log version (RMS (log)), absolute relative error (Abs Rel), squared relative error (Sq Rel), average log10 error (log10) and the accuracy with threshold thr (8 < thr). Readers can
7

---

Table 2: Semantic segmentation results ADE20K. I: inner-product similarity; B: bilinear similarity;
G: gated bilinear similarity. Best performance in boldface, and second best is underlined.
SegFormer B1 [7]
MaskFormer SwinB [8]
Mask2Former SwinB [9]
mloUt
FLOPs
Params
mloUt
FLOPs
Params
mloUt
FLOPs
Params
Nearest
52.70
195
102
Bilinear
41.68
15.91
13.74
53.90
223
107
CARAFE [1]
42.82
+1.45
+0.44
53.53
+0.84
+0.22
53.94
+0.63
+0.07
IndexNet [3]
41.50
+30.65
+12.60
52.92
+17.64
+6.30
54.71
+13.44
+2.10
A2U [5]
41.45
40.41
+0.06
52.73
+0.23
40.03
54.40
40.18
+0.01
SAPA-I
43.05
+0.75
+0.00
53.25
+0.43
+0.00
55.05
+0.33
+0.00
SAPA-B
43,20
+1.02
+0.10
53.15
+0.59
+0.05
54.98
+0.45
+0.02
SAPA-G
44.39
+1.02
+0.10
53.78
+0.59
+0.05
55.22
+0.45
+0.02
Table 3: Object detection results with Faster R-CNN MS COCO, Best performance in boldface, and second best is underlined.
Faster R-CNN [26] (R50)
Params
AP
APso
AP7s
APs
APM
APt
Nearest
41.53
37.4
40.4
21.2
41.0
48.1
CARAFE
+0,22
38.6
59.9
23.3
49.7
IndexNet [3]
+6.30
37.6
58.4
40.9
21.5
41.3
49.2
A2U [S]
+0.03
37.3
58.7
40.0
21.7
41.1
48.5
SAPA-I
+0
37.7
59.2
40.6
22.2
41.2
48.4
SAPA-B B
+0.05
37.8
59.2
40.6
22.4
41.4
49.1
SAPA-G
+0.05
37.8
59.1
40.8
21.5
41.4
48.6
Table Depth estimation results NYU Depth V2 with BTS. Best performance is in boldface.
BTS [11]
accuracy 1
error
R50
Params
8
1.25
5 <. 1.25?
8
1.25
Abs Rel
Sq Rel
RMS
RMSlog
log10
Nearest
49.53
0.865
0.975
0.993
0.119
0.075
0.419
0.152
0.051
CARAFE [1]
+0.41
0.864
0.974
0.994
0.117
0.071
0.418
0.152
0.051
IndexNet [3]
+44.20
0.866
0.976
0.995
0.117
0.071
0.416
0.151
0.050
A2U
+0.08
0.860
0.973
0,993
0.121
0.077
0.429
0.156
0.052
SAPA-B
+0.16
0.871
0.977
0.994
0.116
0.070
0.410
0.151
0.050
SAPA-G
+0.25
0.872
0.978
0.995
0.116
0.069
0.408
0.149
0.049
refer to [11] for the definitions of the metrics. We replace all upsampling stages but the last one for SAPA, due available high-res feature map for the last stage.
For image matting, we train the model on the Adobe Image Matting dataset [13] and report four metrics on the Composition-Ie 1k test set, including Sum of Absolute Difference (SAD), Mean Squared Error (MSE), Gradient (Grad), and Connectivity (Conn) [29]. A2U matting [S] is adopted as the baseline. We use the code provided by the authors and follow the same training settings in the original paper.
Main Results
compare SAPA and its variants against different upsampling operators on the three strong segmentation baselines. Results are shown in Table 2, from which we can see that SAPA consistently outperforms other upsampling operators. Note that SAPA can work well even without parameters and achieves the best performance with only few additional #Params and #FLOPs.

---

Table 5: Image matting results Adobe Composition- 1k data set. Best performance is in boldface.
A2U Matting [5] (R34)
Params
SAD
MSE
Grad.
Conn.
Bilinear
8.05
34.22
0.0090
17.50
31.55
CARAFE [1]
+0.26
32.50
0,0086
16.36
30.35
IndexNet [3]
+12.26
33.36
0.0086
16.17
30.62
A2U [5]
+0.02
32.05
0.0081
15.49
29.21
SAPA-I
34.25
0.0091
18.93
32.09
SAPA-B
+0.04
31.19
0.0079
15.48
28.30
SAPA-G
+0.04
30.98
0.0077
15.59
27.96
Table 6: Ablation studies. I: fnner-produs similarity; B: bilinear similarity; P: plain addition; G:
gated bilinear similarity.
SegFormer
Sim fune
Embedding dim
Setting
I
B
P
G
16
32
64
128
mloU
43.1
43.2
43.2
44.4
43.0
43.2
43.2
43.4
SegFormer
Norm func
Kernel
Sctting
None 41.5
e*
relu 40.8
sigmoid 42.8
softplus 42.7
5
7
mloU
43.2
43.1
43.2
42.5
Results other three dense prediction tasks are shown in Tables 3, 4, and 5. SAPA outperforms other upsampling operators on depth estimation and image matting, but falls behind CARAFE on object detection. One plausible explanation is that the demand of details in object detection is low (Section 6 presents a further in-depth analysis). detail-sensitive tasks like image matting, SAPA significantly outperforms CARAFE. Qualitative comparisons are shown in Fig 4.
5.3 Ablation Study
Here we conduct ablation studies to compare the choices similarity function and normalization function, the effect of different kernel sizes, and the number of embedding dimension. For the default setting, we use the bilinear similarity function, apply the normalization function h(xx) = e", set the kernel size K 5 and the embedding dimension d 32. Quantitative results are shown in Table 6.
Similarity Function We investigate three types of similarity function aforementioned and an additional 'plain addition' baseline. ablates the gating mechanism from the gated bilinear simi- larity. Among them, gated bilinear similarity generates the best performance, which highlights the complementarity semantic regions and boundary details in kernel generation.
Normalization Function We also investigate different normalization functions. 'None' indicates no normalization is used. Among validated functions, h(z) = e" works the best, which means normalization matters. The other three have to play with the epsilon trick prevent zero division.
The kernel size controls the region that each point in the upsampled feature attend to. Results show that, compared with a large kernel, a small local window sufficient to distinguish the semantic clusters.
Kernel Size
Embedding Dimension We further study the influence of embedding dimension in the range of 16, 32, 64, and 128. Interestingly, results suggest that SAPA is not sensitive embedding dimension. This also verifies that SAPA extracts existing knowledge rather than learn unknown knowledge.
6 Discussion
Understanding SAPA from Backward Perspective We have explained SAPA in the forward pass, here we further discuss how may work during training. In fact, originally the model does not know to which the semantic cluster encoder point should belong. The working mechanism

---

Image
similar to Ground Truth
IndexNet
A2U
CARAFE
SAPA
Figure 4: Qualitative results of different upsampling operators on different tasks. From top to bottom, semantic segmentation, object detection, image matting, and depth estimation.
of SAPA assigns each encoder feature point possibility to choose cluster. During training, the ground truths produce gradients, thus changes the assignment possibility. In SAPA, for every encoder point, the semantic clusters in the corresponding local window in decoder features serve implicit labels and cooperate with the ground truths. The correct cluster is the positive label, while the wrong one is negative. In the preliminary stage of training, if encoder feature point is wrong cluster, it will be punished by gradients and engender large losses, and vice versa. Therefore, the encoder feature points can gradually learn its affiliation. We find this process fast by visualizing the epoch-to-epoch feature maps.
Limitations compared with CARAFE CARAFE a purely semantic-driven upsampling operator able to mend semantic information with single-input flow. Such mechanism in CARAFE brings advantages in smoothing semantic regions. E.g., we observe mends holes a continuous region. However, due its single-input flow, cannot compensate the details lost in downsampling. Our SAPA, by contrast, mainly aims to compensate details such as textures and boundaries. SAPA characters in two aspects: semantic preservation and detail delineation. However, Eq. (4) suggests, we do not add any semantic mending mechanism in SAPA. This explains why SAPA is worse than CARAFE on object detection. because detection has less demand for details but more for regional integrity. In short, CARAFE highlights semantic mending, while SAPA highlights semantic preserving and detail delineation.
Conclusion
In this paper, we introduce similarity-awarr point affiliation, i.e., SAPA. It not only indicates lightweight but effective upsampling operator suitable for tasks like semantic segmentation, but also expresses a high-level concept that characterizes feature upsampling. SAPA can serve as an universal substitution for conventional upsampling operators. Experiments show the effectiveness of SAPA and also indicate its limitation: it is more suitable for tasks that favor detail delineation.
Acknowledgement Grant No. 106000
This work is supported by the Natural Science Foundation of China under
10

---

References
[1] Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin. CARAFE: Context-aware reassembly of features. In Proceedings of IEEE International Conference Computer Vision (ICCV), pages 3007-3016, 2019.
[2] Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin. CARAFE++: Unified Aware ReAssembly of FEatures, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[3] Hao Lu, Yutong Dai. Chunhua Shen, and Songcen Xu. Indices matter: Learning index for deep image matting. In Proceedings of IEEE International Conference on Computer Vision (ICCV), pages 3266-3275, 2019.
[4] Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu. Index networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):242-255, 2022.
[5] Yutong Dai, Hao Lu, and Chunhua Shen. Learning affinity-aware upsampling for deep image matting. In Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 6841-6850, 2021.
[6] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), 2017.
[7] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Proceedings of Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Proceedings of Annual Conference on Neural Information Processing Systems (NeurIPS), volume 34, 2021.
[9] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. Proceedings of IEEE Conference Computer Vision Pattern Recognition (CVPR), 2022.
[10] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[11] Jin Han Lee, Myung- Kyu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv Computer Research Repository, 2019.
[12] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In Proceedings of European Conference on Computer Vision (ECCV), pages 746-760, 2012.
[13] Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 2970-2979, 2017.
[14] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481-244 2017.
[15] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene understanding benchmark suite. In Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 567-576, 2015.
[16] Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, and Vivienne Sze. FastDepth: Fast monocular depth estimation on embedded systems. Proceedings of IEFE International Conference Robotics and Automation (ICRA), 2019.

---

[17] Ruibo Li, Ke Xian, Chunhua Shen, Zhiguo Cao. Hao Lu, and Lingxiao Hang. Deep attention- based classification network for robust depth prediction. In Proceedings of Asian Conference on Computer Vision (ACCV), pages 663-678, 2018.
[18] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of IEEE Conference Computer Vision Pattern Recognition (CVPR), pages 1874-1883, 2016.
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4) 2015.
[20] H. Gao, H. Yuan, Z. Wang, and Ji. Pixel transposed convolutional networks. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, pages 1-1, 2019.
[21] Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 1(10), 2016.
[22] Zbigniew Wojna, Vittorio Ferrari, Sergio Guadarrama, Nathan Silberman, Liang-Chieh Chen, Alireza Fathi, and Jasper Uijlings. The devil is in the decoder: Classification, regression and gans. International Journal of Computer Vision, 127(1 16941):66944176. 2019.
[23] Z. Tian, T. He, C. Shen, and Y. Yan. Decoders matter for semantic segmentation: Data-dependent decoding enables flexible feature aggregation. In Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), 2020.
[24] Hamed Pirsiavash, Deva Ramanan, and Charless Fowlkes. Bilinear classifiers for visual recognition. Proceedings of Annual Conference on Neural Information Processing Systems (NeurIPS), 22, 2009.
[25] Hao Lu, Wenze Liu, Hongtao Fu, and Zhiguo Cao. FADE: Fusing the assets of decoder and encoder for agno-sgi upsampling. Proceedings of European Conference on Computer Vision (ECCV), 2022.
[26] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real real-time object detection with region proposal networks. Proceedings of Annual Conference on Neural Information Processing Systems (NeurIPS), 28, 2015.
[27] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 770-778, 2016
[28] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv Computer Research Repository, 2019.
[29] Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. A perceptually motivated online benchmark for image matting, In Proceedings of IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 1826-1833, 2009.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
12

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL.)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] Our method generates stable results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] For each of the datasets used, we cite the creators, shown in Section 5.1.
(b) Did you mention the license of the assets? [Yes] The data used in our work is open source and can be used for academic research.
(c) Did you include any assets either in the supplemental material Our code for this work will appear in the supplementary material.
URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The data used in our work open source and can be used for academic research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The data do not contain personally identifiable information offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "XOsTNBcLwe", "reviewer": "Reviewer_kfqX", "summary": "The paper presents an interesting feature unsampling module, which can be flexible applied to the tasks with upsampling like segmentaiton, detection and depth estimation. The main idea is to generate the kernels based on the feature clustering similarity. It provides extensive experiments on different dense prediction tasks and consistent performance gain has been obtained. The paper is well motivated and the presentation is clear. ", "questions": "1. The segmentation experiments are based on three transformer-based models. How about the segmentation performance for the CNN-based approaches?\n2. The proposed approach relies on the similarity score. It is similar to a new interpolation between the features. Thus, how about is result of the high-frequency data if the ground-truth mask within a  local neighborhood changes abruptly?", "limitations": "The paper claims the potential limitation on the object detections which requires on the semantic mending. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strength:\n1. The idea to design a feature upsampling framework based on the clustering similarity is interesting and novel.\n2. The proposed module obtains state-of-art performance on dense prediction tasks like segmentation, detection, and depth estimation, without large computational overhead.\n3. The proposed module is well motivated and the presentation of the paper is clear. \n\nweakness:\n1. As shown in Table 3, the performance of on object detection is a little bit lower than the baseline of CARAFE [1]. Although the authors provide an explaination in Section 6, usually the better segmentation results should lead to better localization on the bounding-box level. \n2. Compared with the baselines, the implementation of the proposed approach is a little bit complicated with more steps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NKdgQ2bON_1", "reviewer": "Reviewer_jFju", "summary": "This paper introduces a new kernel upsampling module which encourages not only semantic smoothness but also boundary sharpness in order to enhance the performance of semantic segmentation/matting tasks. The key idea is to design a similarity-aware kernel which compare the similarity between encoded features with local spatial awareness about the decoded features. The paper has also proposed a lightweight upsampling operator,  Similarity-Aware Point Affiliation (SAPA), which demonstrated high performance on various dense prediction tasks. The proposed method has compared with CARAFE, IndexNet and A2U in term of both quantitative results and computational complexity. ", "questions": "Although table 1 has summarize the theoretical complexity of various, I would like to see the actual runtime of the proposed method compared with other methods. \n\nBesides the quantitative results in table 2 and table 3, I would like to see some qualitative results which show the boundaries of the segmented regions, especially for the matting tasks. \n\nThe ablation study about the kernel size seems to be too simple, I hope there is a deeper study about its effectiveness.\n", "limitations": "The paper has discussed it properly.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this is a solid paper and I think the proposed method is very useful in dense prediction tasks that require very accurate boundary. The experimental comparisons are also sufficient to validate the performance of the proposed method, and I think its simplicity would make this method useful in real world applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9DdWooU8D17", "reviewer": "Reviewer_W9Gs", "summary": "This paper introduces a point affiliation for feature upsampling which is one of the most essential parts, especially dense prediction networks. The proposed method generates similarity-aware kernels by comparing the similarity between each encoder feature point and the spatially associated local region of decoder features. It also introduce a lightweight upsampling operator, termed Similarity-Aware Point Affiliation (SAPA) and its variant. Experiments show the superiority of the proposed upsampling module on various depth prediction tasks.", "questions": "- Learning feature upsampling is interesting and makes sense. But this framework requires additional learnable parameters, which may make the networks more suffer from the overfitting problem, less generalizable. It would be great if there are experiments about this overfitting issue and generalization issues.\n- In Kernel generation part, the similarity between encoder feature and decoder feature is computed. But, such encoder feature and decoder feature may have different data distributions, so directly computing the similarity between them may be sub-optimal. To overcome this, very simple normalization prior to computing similarity may be used. It would be great if the relevant comments or experiments are additional conducted.\n- The proposed module, SAPA, consists of many sub-modules, e.g., Y embedding, X embedding, Gated addition, Kernel generation, etc. So through experiments for ablation study is required. The current ablation study in Table 4 did not cover all the variants of ablation study. \n\nMinor comments:\n- Using subscript of subscript, e.g., in Line 184, may make the reader follow the paper. It would be better if simpler notations are used.\n- In Similarity Function in Line 203-205, why \"low-rank\" versions are used? Please clarify this.", "limitations": "The paper discussed the limitation of the proposed method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Introducing the notion of point affiliation into feature upsampling is interesting and makes sense. Many other followers will benefit from such notions. \n+ In depth comparison to other previous upsampling methods such as CARAFE or IndexNet, e.g., in Fig. 2, is interesting.\n+ In most cases, the state-of-the-art performance is attained when it is incorporated with various dense prediction networks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3gPGSInzJGX", "reviewer": "Reviewer_4pwd", "summary": "This work proposes a new approach for upsampling decoder features with the guidance of encoder feature, leading to semantic preserving and detail delineation. The key idea is to apply similarity-aware upsampling based on encoder and decoder features. To be more specific, the decoder feature is upsampled via the weighted sum in the sub-region, where the weight is calculated using the similarity between encoder and decoder features, as shown in Figure 3. ", "questions": "- In Figure 3, the decoder feature is added into the encoder feature. This was not explained in the paper.\n\n- The similarity kernel w_{i1, j1, m, n}, which is used for obtaining the upsampled decoder feature at (i1, j1), is calculated with the single encoder feature at (i1, j1) and the set of decoder features (m,n) for m=-r,...,r and n=-r,...,r. This means that upsampling relies on the single encoder feature at (i1, j1), and thus it seems that the proposed method is still too sensitive to the noise of the encoder feature.\n", "limitations": "N.A.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+) The dense prediction tasks (semantic segmenation, depth etimation, and image matting) often require an accurate and detail-preserving upsampling of decoder features, and the proposed simple upsampling scheme can be very effective with no significant computational overhead.\n\n+) Experiments validated the performance gain on the above-mentioned tasks.\n\n-) The overall idea is very simple, and it can be explained using (2) and (3) together with Figure 3. Nevertheless, the manuscript needs to be revised in a more compact form. In the abstract and introduction, authors stated that the point affiliation (semantic cluster) should be incorporated in the upsampling process of decoder features. 'Cluster' seems to be rather misleading. The proposed idea is just a similarity based summation, where the similarity is measured between encoder and decoder features. There is no explicit clustering and affiliation estimation process in the proposed framework.  \n\n-) The equation (4) is rather meaningless. The proposed kernel is rather similar to joint bilateral filter, and it is well-known that in the smooth region, the joint bilateral kernel becomes the Gaussian function. Though the proposed kernel becomes a uniform function, such a derivation seems unnecessary in the main paper (maybe it can be moved to supplementary material).\n\n**********************************************************************************\n* After rebuttal\n\nAuthors addressed some questions in the rebuttal. I appreciate it.\n\nThis work introduces a simple yet effective approach for dense prediction tasks. This kind of upsampling (using two inputs) has been adopted in various vision tasks, but using it in the upsamling process for decoder features sounds interesting.\nNevertheless, considering the simplicity of the overall method, I would like to keep the initial rating (Borderline accept).\n**********************************************************************************\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept