LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention

Authors: zhenjiang  hu
guiyu  zhao
ying  jin
hongbin  ma
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 11:27:06

Abstract: Feature upsampling is an essential operation in constructing deep convolutional neural networks. However, existing upsamplers either lack specific feature guidance or necessitate the utilization of high-resolution feature maps, resulting in a loss of performance and flexibility. In this paper, we find that the local self-attention naturally has the feature guidance capability, and its computational paradigm aligns closely with the essence of feature upsampling (\ie feature reassembly of neighboring points). Therefore, we introduce local self-attention into the upsampling task and demonstrate that the majority of existing upsamplers can be regarded as special cases of upsamplers based on local self-attention. Considering the potential semantic gap between upsampled points and their neighboring points, we further introduce the deformation mechanism into the upsampler based on local self-attention, thereby proposing LDA-AQU. As a novel dynamic kernel-based upsampler, LDA-AQU utilizes the feature of queries to guide the model in adaptively adjusting the position and aggregation weight of neighboring points, thereby meeting the upsampling requirements across various complex scenarios. In addition, LDA-AQU is lightweight and can be easily integrated into various model architectures. We evaluate the effectiveness of LDA-AQU across four dense prediction tasks: object detection, instance segmentation, panoptic segmentation, and semantic segmentation. LDA-AQU consistently outperforms previous state-of-the-art upsamplers, achieving performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the baseline models in the aforementioned four tasks, respectively.

Main paper: 
 introduction: 
 As a fundamental operator in deep convolutional neural networks, feature upsampling is widely utilized in various dense prediction tasks, including object detection, semantic segmentation, and image inpainting, etc. Given the spatial downsampling characteristics of convolutional and pooling operators, feature upsampling emerges as a crucial inverse operation, indispensable for meeting task-specific requirements. For instance, it facilitates the restoration of spatial resolution in pixel-level dense prediction tasks and enables multi-scale feature fusion in feature pyramid network (FPN) [18] .Commonly used upsamplers, such as Nearest Neighbor Interpolation and Bilinear Interpolation, aggregate features from neighboring points in a manually designed paradigm, making it difficult to address the requirements of various upsampling tasks simultaneously. Subsequently, several learnable upsampling methods have been proposed, including deconvolution [23] and Pixel Shuffle [32] , etc. However, these methods typically learn a fixed set of parameters for the upsampling kernel, applying the same operation to all spatial positions of the input feature map, leading to suboptimal upsampling results. To enhance the dynamic adaptability of the upsampling operator and enable it to address various upsampling tasks in complex scenes, dynamic filter-based upsamplers have been proposed [26, 35, 41] . However, these methods either lack specific feature guidance or require the intervention of high-resolution images, limiting their application scenarios and performance.Most existing upsampling operators can be viewed as a weighted aggregation of features within local neighborhoods surrounding upsampled points (i.e., feature reassembly). We observe that this is consistent with local self-attention, which determines attention weights and extracts contextual information from uniform neighboring points. However, local self-attention naturally incorporates the query-guided mechanism, aligning well with the upsampling task. This involves adaptively aggregating neighborhood features based on the attributes of the upsampled points, resulting in explicit point affiliations [26] . These insights inspire us to integrate local self-attention into the upsampling task, achieving adaptive upsampling with query guidance in a single layer.In this paper, we introduce a method for incorporating local selfattention into feature upsampling tasks. Additionally, we note that using fixed, uniform neighboring points may lead to suboptimal upsampling result. As depicted in Figure 2 , uniform neighboring point selection in feature maps with high downsampling strides may result in notable semantic disparities, hindering high-resolution feature map generation. To address this, we introduce the deformation mechanism to dynamically adjust the positions of neighboring points based on the features of query points (i.e., upsampled points) and their contextual information, aiming to further enhance the model's adaptability. Based on above, we have named our method as LDA-AQU, which offers the following advantages compared to other dynamic upsamplers: 1) operates on a single layer without requiring high-resolution inputs; 2) possesses query-guided capability, enabling the interactive generation of dynamic upsampling kernels using the features of the query points and their neighboring points; 3) exhibits local deformation capability, permitting dynamic adjustment of positions to neighboring points based on the contextual information of query points. These properties enable LDA-AQU to achieve superior performance while remaining lightweight.Through extensive experiments conducted on four dense predcition tasks including object detection, semantic segmentation, instance segmentation, and panoptic segmentation, we have validated the effectiveness of LDA-AQU. For instance, LDA-AQU can obtain +1.7 AP gains for Faster R-CNN [31] , +1.5 AP gains for Mask R-CNN [11] , +2.0 PQ gains for Panoptic FPN [14] on MS COCO dataset [19] . On the semantic segmentation task, LDA-AQU also brings +2.5 mIoU gains for UperNet [36] on ADE20K dataset [40] . On the aforementioned four dense prediction tasks, our LDA-AQU consistently outperforms the previous state-of-the-art upsampler while maintaining a similar FLOPs and parameters. 
related work 2.1 feature upsampling: 
 Commonly used upsamplers, such as Nearest Neighbor Interpolation and Bilinear Interpolation, are popular for their simplicity and efficiency in various visual tasks. To enhance the adaptability, some learnable upsamplers have been proposed. Deconvolution [23] employs a reverse convolution operation to achieve the upsampling of feature maps. Pixel Shuffle (PS) [32] increases the spatial resolution of the feature map by shuffling features along both spatial and channel directions. With the popularity of dynamic networks [3, 38, 41] , some upsamplers based on dynamic kernels have been introduced. CARAFE [35] utilizes a content-aware approach to generate dynamic aggregate weights. IndexNet [24] models various upsampling operators as different index functions and proposes several index networks to generate indexes for guiding upsampling. SAPA [26] introduces the concept of point membership into feature upsampling and guides kernel generation through the similarity between semantic clusters. In comparison, our LDA-AQU utilizes the features of the query point to generate deformed offsets and aggregate weights of neighboring points, achieving dynamic feature upsampling with feature guidance from a single low-resolution input.Dense prediction involves pixel-level tasks such as object detection [16, 30, 31] , instance segmentation [1, 9, 11] , panoptic segmentation [14, 15, 17] , and semantic segmentation [4, 23, 37] . Models for such tasks typically include a backbone network [12, 13, 33] , a feature pyramid network [10, 18, 20] , and one or more task heads. The backbone network reduces data dimensions and extracts salient features to decrease computational complexity and capture robust semantic information. The feature pyramid network connects multiscale features to enhance the model's perception across scales. The task head links extracted features to the prediction task, serving as the primary distinction among different models.For instance, Faster R-CNN [31] uses a detection head for object recognition and localization tasks. Mask R-CNN [11] achieves both object detection and instance segmentation by adding an instance segmentation head based on Faster R-CNN. Similarly, Panoptic FPN [14] integrates semantic segmentation, instance segmentation, and panoptic segmentation tasks by incorporating an additional semantic segmentation head based on Mask R-CNN. UperNet [36] uniformly conducts scene perception and parsing by integrating various task heads. Due to the spatial reduction characteristics of convolution and pooling operations, feature upsampling becomes essential for accomplishing above dense prediction tasks. Our LDA-AQU can be easily integrated into these frameworks and consistently brings stable performance improvements.As vanilla self-attention in Vision Transformer (ViT) [8] capture global contextual information through dense interactions, the computational complexity and memory usage become unbearable for high-resolution inputs. Therefore, in recent years, numerous studies have aimed to optimize the efficiency of ViT. Swin Transformer [22] decreases interactive tokens by confining them to non-overlapping windows, and then captures global dependencies through window sliding. CSwin Transformer [7] further improves efficiency with a cross-shaped interactive window. Additionally, some methods based on local self-attention [27, 29, 34, 39] have been proposed to optimize model efficiency and introduce the inductive bias of convolution. We incorporate local self-attention into the upsampling task and introduce the deformation mechanism for neighboring points to enhance the model's dynamic adaptability. 
method: 
 First, we will provide a brief overview of self-attention and local selfattention. Then, we will elaborate on our approach for extending the local self-attention to address feature upsampling tasks, which we refer to as LA-AQU. Finally, we will introduce the integration of the deformation mechanism into LA-AQU, presenting LDA-AQU as a means to enhance its adaptability in complex scenarios.(𝒙𝒙, 𝒚𝒚) (𝒙𝒙𝒙, 𝒚𝒚𝒙) Upsample (𝒙𝒙 𝟏𝟏 , 𝒚𝒚 𝟏𝟏 ) (𝒙𝒙 𝟐𝟐 , 𝒚𝒚 𝟏𝟏 ) (𝒙𝒙 𝟏𝟏 , 𝒚𝒚 𝟐𝟐 ) (𝒙𝒙 𝟐𝟐 , 𝒚𝒚 𝟐𝟐 ) (𝒙𝒙𝒙, 𝒚𝒚𝒙) (𝒙𝒙 𝟐𝟐 , 𝒚𝒚 𝟐𝟐 )(𝒙𝒙 𝟑𝟑 , 𝒚𝒚 𝟐𝟐 ) (𝒙𝒙 𝟐𝟐 , 𝒚𝒚 𝟑𝟑 ) (𝒙𝒙 𝟑𝟑 , 𝒚𝒚 𝟑𝟑 ) (𝒙𝒙𝒙, 𝒚𝒚𝒙) (𝒙𝒙 𝟏𝟏 , 𝒚𝒚 𝟐𝟐 ) (𝒙𝒙 𝟏𝟏 , 𝒚𝒚 𝟑𝟑 ) (𝒙𝒙 𝟏𝟏 , 𝒚𝒚 𝟏𝟏 )(𝒙𝒙𝟐𝟐, 𝒚𝒚 𝟏𝟏 ) (𝒙𝒙 𝟑𝟑 , 𝒚𝒚 𝟏𝟏 ) Aligned Corner Points Upsampled Point Neighbors of Upsampled Point (a) (b)𝑿𝑿 𝒀𝒀 3.1 Preliminary3.1.1 Self-Attention. Given a flattened input feature map 𝑥 ∈ R 𝑁 ×𝐶 ,where 𝐶 is the channel size and 𝑁 = 𝐻 ×𝑊 is the number of tokens along the spatial dimension, the output of the 𝑖-th token 𝑧 𝑖 after standard self-attention can be expressed as:EQUATIONEQUATIONwhere 𝑊 𝑞 ,𝑊 𝑘 ,𝑊 𝑣 ∈ R 𝐶 ×𝐶 are the linear projection matrices. For simplicity, we ignore the output projection matrix 𝑊 𝑜 and the normalization factor 𝑑 𝑘 , while also fixing the number of heads to 1. The standard self-attention employs dense interaction for each query to gather crucial long-range dependencies. Hence, the computational complexity can be expressed as O (2𝑁 2 𝐶 + 4𝑁𝐶 2 ), with O (2𝑁 2 𝐶) for dense interaction and O (4𝑁𝐶 2 ) for linear projection.3.1.2 Local Self-Attention. Since the computational complexity scales quadratically with the number of tokens, researchers are exploring the use of local self-attention, aiming to reduce computational complexity and introduce the local induction bias of convolution. Assuming the kernel size of the neighborhood sampling is 𝑛. For a flattened input feature map 𝑥 ∈ R 𝑁 ×𝐶 , the output of the 𝑖-th token 𝑧 𝑖 after local self-attention can be expressed as:EQUATIONEQUATION)where q ∈ R 𝑁 ×1×𝐶 denotes the queries after the 𝑅𝑒𝑠ℎ𝑎𝑝𝑒 operation. 𝜙 (•, •) is the sampling function, which can be easily accomplished by the built-in funcion 𝑢𝑛𝑓 𝑜𝑙𝑑 in PyTorch [28] . The k, ṽ ∈ R 𝑁 ×𝑛 2 ×𝐶 are the keys and values after neighborhood sampling, respectively. The local self-attention restricts feature interaction to the local neighborhood of each query, thus the computational complexity LayerNorm GELU 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 Conv 𝒌𝒌 𝒆𝒆 × 𝒌𝒌 𝒆𝒆 𝐻𝐻 × 𝑊𝑊 × 2𝑘𝑘 𝑢𝑢 2 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝛼𝛼𝐻𝐻 × 𝛼𝛼𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝐻𝐻 × 𝑊𝑊 × 𝐶𝐶 𝛼𝛼 2 𝐻𝐻𝑊𝑊 × 1 × 𝐶𝐶 𝛼𝛼 2 𝐻𝐻𝑊𝑊 × 𝑘𝑘 𝑢𝑢 2 × 𝐶𝐶 𝛼𝛼 2 𝐻𝐻𝑊𝑊 × 𝑘𝑘 𝑢𝑢 2 × 𝐶𝐶 𝛼𝛼𝐻𝐻 × 𝛼𝛼𝑊𝑊 × 𝐶𝐶 PE Figure 4:Assuming the kernel size of neighborhood sampling is 𝑘 𝑢 = 3. As shown in Figure 3 (b), let 𝑝 = (𝑥, 𝑦) denote the coordinate of the point to be interpolated, where 𝑥 ∈ [0,𝑊 -1] and 𝑦 ∈ [0, 𝐻 -1]. Taking the grid arrangement format of aligned corner points as an example, we can obtain the corresponding coordinate 𝑝 ′ = (𝑥 ′ , 𝑦 ′ ) in the input feature map byEQUATIONLet 𝑟 = {(𝑥 1 , 𝑦 1 ), (𝑥 2 , 𝑦 1 ), ..., (𝑥 3 , 𝑦 3 )} denote the absolute coordinates of uniform neighboring points of 𝑝 ′ . The upsampled result of point 𝑝 based on the local self-attention can be expressed as:EQUATIONLet 𝐹 (𝑝 ′ , 𝑠) = exp(𝑄 (𝑝 ′ )𝐾 (𝑠 ) T )𝑡 ∈𝑟 exp(𝑄 (𝑝 ′ )𝐾 (𝑡 ) T ) . When 𝑊 𝑉 is an identity matrix, the above formula can be further simplified toEQUATIONConsidering the properties of 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥, we have 𝑠 ∈𝑟 𝐹 (𝑝 ′ , 𝑠) = 1. Therefore, LA-AQU can be described as the adaptive acquisition of aggregation weights guided by the query features and the reassembly of features from neighboring points. Indeed, many existing upsamplers can be formulated by upsampler based on local self-attention. We will delve into this concept in Section 3.3.Mechanism. LA-AQU employs a fixed and uniform neighborhood sampling scheme for feature upsampling. Consequently, it becomes challenging to meet the upsampling needs across various complex scenarios simultaneously. As shown in Figure 2 , solely aggregating the features of uniformly sampled neighboring points will result in the model overly emphasizing less significant background regions while neglecting the features of the object itself. Therefore, we further introduce the deformation mechanism and propose LDA-AQU.The overall process of LDA-AQU is illustrated in Figure 4 . Similarly, considering the input feature map 𝑋 ∈ R 𝐻 ×𝑊 ×𝐶 and the output feature map 𝑌 ∈ R 𝛼𝐻 ×𝛼𝑊 ×𝐶 , we initially obtain the 𝑄, 𝐾, 𝑉 through Equation 6. To avoid extra dimensional transformations and reduce the number of parameters, we employ bilinear interpolation to upsample the matrix 𝑄 ∈ R 𝐻 ×𝑊 ×𝐶 , resulting in 𝑄 ′ ∈ R 𝛼𝐻 ×𝛼𝑊 ×𝐶 . Then, utilizing the built-in function 𝑚𝑒𝑠ℎ𝑔𝑟𝑖𝑑 in PyTorch, we can generate the uniform coordinate matrix 𝑃 ∈ R 𝛼𝐻 ×𝛼𝑊 ×1×2 for the upsampled feature map. Through Equation 7, we can derive the reference point coordinate matrix 𝑃 ′ by projecting 𝑃 onto the input feature map.Assuming the kernel size of neighborhood sampling is 𝑘 𝑢 , the initial offset 𝛥𝑃 of neighboring points can be obtained asEQUATIONSubsequently, we can obtain the coordinate matrix of neighboring points 𝑅 ∈ R 𝛼𝐻 ×𝛼𝑊 ×𝑘 2 𝑢 ×2 of the upsampled points through the broadcast mechanism by 𝑅 = 𝑃 ′ + 𝛥𝑃.To enable the neighboring points to dynamically adjust their positions, we introduce a sub-network 𝜁 (•), utilizing query features to generate the query-guided sampling point offset matrix 𝛥𝑅 ∈ R 𝛼𝐻 ×𝛼𝑊 ×𝑘 2 𝑢 ×2 . Based on the uniform neighboring point coordinate matrix 𝑅 and the predicted offset matrix 𝛥𝑅, the final deformed neighboring point coordinate matrix 𝑅 ′ can be obtained byEQUATIONFinally, we utilize the point-wise features of matrix 𝑄 ′ to interact with the corresponding local point features of 𝐾, 𝑉 to complete the upsampling task. The aforementioned process can be expressed as:EQUATIONEQUATIONwhere 𝑄 ∈ R 𝛼𝐻 ×𝛼𝑊 ×1×𝐶 denotes the queries after 𝑅𝑒𝑠ℎ𝑎𝑝𝑒 operation. 𝐾, 𝑉 ∈ R 𝛼𝐻 ×𝛼𝑊 ×𝑘 2 𝑢 ×𝐶 are the keys and values after neighboring sampling. To ensure differentiability, we use bilinear sampling function Φ(•, •) for sampling with non-integer offsets.Offset Predictor. The detailed structure of the offset predictor 𝜁 (•), utilized to generate the sampling point offset matrix 𝛥𝑅, is depicted in Figure 4 . We employ a 3×3 depthwise convolution to extend the perceptual range of the queries and utilize a convolutional layer with the kernel size of 𝑘 𝑒 to predict the offset for 𝑘 2 𝑢 points. The ablation studies of 𝑘 𝑒 and 𝑘 𝑢 can be found in Section 4.6.Offset Groups. To enhance the model's capability in perceiving distinct channel features and adaptability across diverse scenarios, we partition the channels of the input feature maps within the offset predictor 𝜁 (•), employing varied offsets for different groups.Local Deformation Ranges. To expedite convergence speed, we apply the 𝑇 𝑎𝑛ℎ function to the results of the Offset Predictor, limiting them to the range [-1, 1]. Additionally, we employ a factor 𝜃 to regulate the deformation range of the neighboring points. In our experiments, we observe that increasing the value of 𝜃 will bring some performance improvements. We believe that employing a larger local deformation range, combined with the grouping operation, will enable the model to capture a wider range of crucial features within the neighborhood for refining the upsampling results. More detailed analysis will be conducted in Section 4.6.In this section, we will explore the relationship between different upsamplers and LDA-AQU. Indeed, LA-AQU can already represent the majority of upsampling schemes.Bilinear Interpolation. Similarly, let 𝑋 ∈ R 𝐻 ×𝑊 ×𝐶 and 𝑌 ∈ R 𝛼𝐻 ×𝛼𝑊 ×𝐶 denote the input and output feature map, respectively. As shown in Figure 3(a) , after projection via Equation 7, the feature vector of the upsampled point 𝑝 = (𝑥, 𝑦) can be expressed asEQUATIONwhere 𝑟 = {(𝑥 1 , 𝑦 1 ), (𝑥 1 , 𝑦 2 ), (𝑥 2 , 𝑦 1 ), (𝑥 2 , 𝑦 2 )} is the set of standard neighbor points of point 𝑝 ′ . The bilinear interpolation kernel 𝐹 (𝑝 ′ , 𝑠) in here can be represented asEQUATIONwhere 𝑤 (𝑎, 𝑏) = max(0, 1 -|𝑎 -𝑏 |) represents the aggregation weight based on the distance between point pairs. By comparing Equation 14and Equation 9, we can conclude that upsampling through bilinear interpolation is indeed a special instance of LA-AQU. When the aggregate weight of points {(𝑥 1 , 𝑦 1 ), (𝑥 2 , 𝑦 1 ), (𝑥 3 , 𝑦 1 ), (𝑥 1 , 𝑦 2 ), (𝑥 1 , 𝑦 3 )} of 𝐹 (𝑝 ′ , 𝑠) in Equation 9equals zero, and the computation results are based on distance, LA-AQU will degrade into upsampling based on bilinear interpolation.CARAFE. CARAFE and LA-AQU both compute dynamic contentaware aggregation weights for neighboring points of upsampled points. The distinction lies in LA-AQU using query-related dynamic features to interactively generate kernel weights, whereas CARAFE uses a convolutional layer. Moreover, the query-guided weight generation in LA-AQU enables the model to prioritize features that closely align with its own content.DySample. DySample achieves feature upsampling through sampling. With the deformation mechanism, LDA-AQU achieves similar effects but employs the query-guided mechanism for neighboring deformation and feature aggregation. Compared to DySample, LDA-AQU leverages object-specific features effectively, and the kernel-based upsampling scheme aligns better with human intuition, i.e., using neighboring points for feature inference.It is noteworthy that LDA-AQU avoids using the PixelShuffle operator, in contrast to CARAFE and DySample, enabling it to achieve any desired multiple of feature upsampling.Given an input feature map with the shape of 𝐻 × 𝑊 × 𝐶, the overall computational complexity of LDA-AQU can be expressed asO (2𝐻𝑊 𝐶 2 + 2𝛼 2 𝑘 2 𝑢 𝐻𝑊 𝐶 + 2𝛼 2 𝑘 2 𝑢 𝑘 2 𝑒 𝐻𝑊 𝐶), with O (2𝐻𝑊 𝐶 2 ) for linear projection, O (2𝛼 2 𝑘 2𝑢 𝐻𝑊 𝐶) for attention interaction, and O (2𝛼 2 𝑘 2 𝑢 𝑘 2 𝑒 𝐻𝑊 𝐶) for deformed offsets prediction. Note that we ignore the computational complexity of depthwise convolution and positional encoding as they are considerably lower than the aforementioned blocks. To summarize, LDA-AQU exhibits linear computational complexity with the number of input tokens. 
experiments 4.1 experimental settings: 
 We evaluate the effectiveness of proposed LDA-AQU on four challenging tasks, including object detection, instance segmentation, semantic segmentation, and panoptic segmentation.Datasets and Evaluation Metrics. We utilize the challenging MS COCO dataset [19] to evaluate the effectiveness of LDA-AQU across object detection, instance segmentation, and panoptic segmentation tasks, respectively. For object detection and instance segmentaion tasks, we report the standard COCO metrics of Mean Average Precision (mAP). For the panoptic segmentation task, we report the PQ, SQ, and RQ metrics [15] as in Dysample [21] . For the semantic segmentation task, we conduct performance comparison using the ADE20K dataset [40] and report the Average Accuracy (aAcc), Mean IoU (mIoU), Mean Accuracy (mAcc) metrics.Implementation Details. We evaluate the effectiveness of LDA-AQU using MMDetection [2] and MMSegmentation [5] toolboxes. Specifically, we adopt Faster R-CNN [31] , Mask R-CNN [11] , Panoptic FPN [14] and UperNet [36] as the baseline models. If unspecified, the offset groups, local deformation ranges, and channel size reduction factors are set to 2, 11, and 4, respectively. We use 1× training schedule for object detection, instance segmentation and panoptic segmentation tasks. For semantic segmentation, the model is trained for 160K iterations. All other training strategies and hyperparameters remain the same as in [21] for fair comparison. [32] ResNet-50 37.5 58.5 40.4 21.5 41.5 48.3 +9.4M +50.2G CVPR16 CARAFE [35] ResNet-50 38.6 59.9 42.2 23.3 42.2 49.7 +0.3M +1.6G ICCV19 IndexNet [24] ResNet-50 37.6 58.4 40.9 21.5 41.3 49.2 +8.4M +46.4G ICCV19 A2U [6] ResNet-50 37. As shown in As illustrated in As shown in As shown in Table 4 , by replacing the upsamplers with LDA-AQU in the FPN and Multi-level Feature Fusion (FUSE) of UperNet, the mIoU of baseline model has been improved from 39.78 to 42.31, surpassing CARAFE by 1.31 mIoU and DySample by 1.23 mIoU. In addition, we also report the aAcc and mAcc metrics of the model. As depicted in Table 4 , the performance of LDA-AQU remains superior on these two metrics compared to other upsamplers.We conduct ablation studies on MS COCO using Faster R-CNN and Mask R-CNN to verify the impact of hyperparameters in LDA-AQU.Local Deformation Ranges. Initially, we assess the impact of 𝜃 in the FPN of Faster R-CNN. As shown in Table 5 , setting 𝜃 of each LDA-AQU to 11 achieves optimal performance (39.2 AP). As 𝜃 decreases, the performance of the model gradually diminishes. We believe the reason is that a few neighboring points are enough to cover sufficient information to ensure accurate interpolation. Therefore, the model prefers to find broader contextual information as auxiliary items to optimize the upsampling results. Then, we verify the influence of 𝜃 in the mask head of Mask R-CNN. As shown in Table 6 , when 𝜃 of mask head is set to 5, the model achieves the best performance. The reason is that the size of the input feature map of the mask head is 14 × 14, so using a larger 𝜃 will make it difficult for the model to focus on local details.Offset Groups. We evaluate the impact of the offset groups on model performance. As shown in Table 7 , the model achieves the optimal performance when the number of groups is set to 2. Excessive groupings will lead to a decreased size of features utilized for predicting offsets, thus impeding model learning.Channel Size Reduction Factors. Finally, we evaluate the impact of channel reduction factors on model performance. As 8 , when the reduction factor is set to 16, LDA-AQU yields an AP gain of 1.4 for Faster R-CNN (38.9 AP v.s 37.5 AP). By reducing the channel reduction factor, the performance of the model gradually improved, finally reaching 39.4 AP. In order to balance the performance and computational complexity of the model, we set the channel reduction factor of LDA-AQU to 4.In this section, we provide some visualizations and analysis of deformed neighbor points and results across different tasks. More visualizations can be found in the supplementary materials. Deformed Neighboring Points. We visualize the locations of some upsampled points and their corresponding deformed neighboring points. As shown in Figure 5 , LDA-AQU can adaptively adjust the position of neighbor points according to the query features (e.g., the fork in the fourth column). Even in occlusion scenes (e.g., third column of Figure 5 ), LDA-AQU demonstrates a tendency to prioritize the featrues of object itself over the occluder (i.e., branches).Qualitative Experiments. We also conduct qualitative experiments to verify the effectiveness of LDA-AQU. Specifically, we visualize the the results of the baseline models (e.g., Faster R-CNN, Mask R-CNN, etc. ) and LDA-AQU across various visual tasks. As shown in Figure 6 , our model exhibits superior performance, thereby validating the effectiveness of LDA-AQU. 
conclusion: 
 In this paper, we introduce local self-attention into the upsampling task. Compared with previous methods, the upsampling based on local self-attention (LA-AQU) naturally incorporates the feature guidance mechanism without necessitating high-resolution input. Additionally, to enhance the adaptability of LA-AQU to complex upsampling scenarios, we further introduce the query-guided deformation mechanism and propose LDA-AQU. Finally, LDA-AQU can dynamically adjust the location and aggregation weight of neighboring points based on the features of the upsampled point. Through extensive experiments on four dense prediction tasks, we evaluate the effectiveness of LDA-AQU. Specifically, LDA-AQU has consistently demonstrated leading performance across above tasks, while maintaining a comparable FLOPs and parameters. For future work, we intend to explore dynamic local deformation ranges and investigate additional application scenarios, including image restoration, image inpainting, downsampling, etc. 
: 
 Figure 1 : Comparison of various upsamplers in terms of network parameters, Mean Average Precision (mAP) and FLOPs (indicated by area of circles) using Faster R-CNN [31] with ResNet-50 [12] as the baseline model. 
 
 Appendix:  


Decision: The initial reviews were mixed and the reviewers raised questions about the implementation details. The authors' rebuttal addressed most of the reviewers' concerns. Now all reviewers lean towards the acceptance of the paper.

Reviews: review: paper_summary: paper_summary: This manuscript introduces a novel feature upsampling method called LDA-AQU, which leverages a local self-attention mechanism along with a query-guided deformation mechanism to adaptively adjust the positions and aggregation weights of neighboring points, thereby effectively enhancing the performance of feature upsampling in deep convolutional neural networks. In dense prediction tasks such as object detection, instance segmentation, panoptic segmentation, and semantic segmentation, LDA-AQU has demonstrated competitive performance while maintaining comparable computational load and parameter count.
strength_weakness: strength_weakness: The manuscript introduces an innovative upsampling method called LDA-AQU, which enhances the performance and flexibility of upsampling by integrating a local self-attention mechanism with a query-guided deformation mechanism. The authors have not only proposed a novel approach but also delved deeply into the principles behind it, including the incorporation of local self-attention and the design of the deformation mechanism. Additionally, LDA-AQU maintains a computational load and parameter count comparable to existing methods, suggesting that it offers high efficiency and scalability in practical applications.
limitations: limitations: 1, In Figure 6, the authors demonstrate a comparison between LDA-AQU and a baseline approach, but I haven't come across a description of this baseline approach within the paper, which leaves me uncertain as to what it is being compared with.

2, In Section 3.2.2, the authors mention, "To avoid continuous projection and sampling, we employ bilinear interpolation to upsample the matrix...," but I don't understand the cause-and-effect relationship here; more explanation from the authors would aid the readers' comprehension.

3, In Tables 1, 2, 3, and 4, the authors present comparative results of LDA-AQU and other methods on various dense prediction tasks. I suggest that the authors could consider including more recent approaches in their comparison, such as "Learning To Zoom and Unzoom" from CVPR 2023.

4, Regarding the comparative methods, I've noticed that the experiments conducted by the authors were all based on convolutional networks. To substantiate the effectiveness of the method and further enhance the persuasiveness of the paper, I recommend that the authors also perform some comparative experiments on Transformer-type networks.

5, In the ablation study section, the authors have conducted detailed experiments on the values of various hyperparameters, but it seems that they have overlooked the ablation experiments on the module structure. Why is depthwise convolution (DWConv) used in some places while regular convolution is used in others? Why was bilinear interpolation chosen over other methods? How can it be determined that the current module design is optimal?
paper_summary: paper_summary: This paper finds existing upsamplers either lack specific feature guidance or necessitate the utilization of high-resolution feature maps. Hence, the authors propose a novel dynamic kernel-based upsampler, LDA-AQU, which introduces local self-attention and deformation mechanism into the upsampling task.  LDA-AQU utilizes the feature of queries to guide the model in adaptively adjusting the position and aggregation weight of neighboring points. LDA-AQU is also lightweight and can be easily integrated into various model architectures. LDA-AQU is evaluated across four dense prediction tasks: object detection, instance segmentation, panoptic segmentation, and semantic segmentation.
strength_weakness: strength_weakness: 1. This paper proposes a novel dynamic upsampler which incorporates local self-attention into feature upsampling.
2. The proposed LDA-AQU has demonstrated effectiveness on various tasks and datasets.
3. The authors have provided comprehensive comparison between the proposed LDA-AQU and existing upsamplers.
4. The exposition of equations in the paper is remarkably clear, and the figures and visualizations are beautifully crafted.
limitations: limitations: 1. LDA-AQU introduces self attention and deformable convolution into upsampling. Hence, LDA-AQU appears quite similar to existing deformable attention [1]. Please clarify the difference. 
2. The authors adopt model parameters and FLOPS to evaluate the computational efficiency, which is misleading. For example, deformable convolution results in irregular neighbors, which makes the memory accessing pattern discontinuous and then increase the memory accessing time. However, FLOPS and parameters cannot adequately reflect such difference. Please provide execution latency and throughput for the evaluation in Figure 1 and Table 1.
3. The comparison in Figure 1 is confusing. Are the parameters and FLOPS shown in the figure for the entire network or only for the upsampler module? If it's the former, then is it ensured that the rest of the network remain consistent during the comparison?
4. The authors' experiments demonstrate that feature-guided upsampling achieves better performance across multiple tasks. However, I am curious about the underlying physical significance. Could an explanation be provided?
5. In line 532 "convolutional layer" rather than "convolutional alyer"

[1] Zhu X, Su W, Lu L, et al. Deformable DETR: Deformable Transformers for End-to-End Object Detection[C]//International Conference on Learning Representations. 2020.

Overall, I acknowledge the work presented in this paper, but further explanations and experiments are needed.
paper_summary: paper_summary: This paper introduces a novel dynamic kernel-based upsampler, LDA-AQU, which utilizes query features to guide the model in adaptively adjusting the positions and aggregation weights of neighboring points across various complex scenarios to meet the upsampling requirements. Experimental results demonstrate that LDA-AQU consistently outperforms previous state-of-the-art upsamplers in four dense prediction tasks and also achieves significant performance improvements.
strength_weakness: strength_weakness: 1. The idea of incorating local self-attention into feature upsampling tasks is interesting and quite simple, and the subsequent proposed deformation mechanism bassed on queries and contexts can adjust the position to enhance the performance.
2. This paper has also shown the relationship between the proposed method and other well-known upsampling methods, which demonstrates the generality and dynamic scheme of LDA-AQU.
3. The experiments on different tasks consistently outperforms competing methods on all tested benchmarks. Several ablation studies are carried out to validate the functionality of hyperparameters.
limitations: limitations: 1. Could the authors clarify the relationship between the factor $\theta$ and the offset range? Providing the formulation or equation would help illustrate this relationship.
2. Have the authors investigated the impact of the upsampling factor $\alpha$ on the performance of the proposed LDA-AQU? Conducting ablations on this factor would provide valuable insights.
3. It would be helpful to explain the rationale behind using $I_v$ instead of $W_v$ in the value branch depicted in Figure 4.
4. Which padding modes are employed when points are located on the image edge? Additionally, does this choice of padding modes affect the performance of LDA-AQU? Clarification on this aspect would be beneficial.