# CenterInst: Center-Based Real-Time Instance Segmentation
## 3 Methods
### Dual-Path Decoder
#### 3.3.1 Center-Guided Sampling-Based Query Decoder  
Numerous studies [21; 24; 25] have demonstrated that the introduction of multiscale features can significantly enhance the segmentation performance of instance segmentation models. However, in real-time instance segmentation tasks, for many methods, such as SparseInst and FastInst, to meet the requirements of real-time processing, the utilization of multiscale features in the decoder is often discarded due to constraints on inference time and computational costs. These methods employ a full-pixel attention mechanism in the decoding process. Their computational costs are heavy, as they require attention to all pixel features on the feature map for each instance query. Nevertheless, the use of full-pixel attention not only leads to an increase in computational complexity but may also compromise accuracy. For FastInst, this issue was identified, but, despite attempting to improve performance by reducing irrelevant receptive fields through the addition of masked attention [4], computational costs were not successfully reduced.  
To further enhance model performance and alleviate computational burdens, we propose a center-guided sampling-based query decoder. By generating sampling points near the instance center, the number of sampling points is significantly reduced, which effectively reduces computational costs. This approach not only enables the generation of specific sparse sampling distributions for different queries, but also avoids the introduction of excessive irrelevant sampling points, thereby improving model accuracy.  
The center-guided sampling-based query decoder is primarily composed of three parts: center-guided adaptive sampling, adaptive mixing, and self-attention. In the initial stage \(S_{0}\), the input consists of the query proposal \(Q_{0}\) and the center position proposal output by the query selector. In the subsequent iterative process, the content queries \(Q_{l-1}\) and center positions \(C_{l-1}\) updated in the previous stage are utilized.  
Center-guided adaptive sampling: We employed a linear layer to adaptively generate a set of sampling offsets \(\left\{\left(\Delta x_{ij},\Delta y_{ij}\right)\right\}\in\mathbb{R}^{3\times N _{point}\times N_{hard}\times 2}\) for each layer's feature map. Similar to multihead attention, \(N_{point}\) and \(N_{head}\) represent the number of sampling points per decoding head and the number of decoding heads, respectively. The sampling point coordinates were obtained by summing the offsets with the instance center position, denoted as \(P_{ij}\in\mathbb{R}^{3\times N_{point}\times N_{head}\times 2}\), where \(i\) and \(j\), respectively, denote the indices of the sampling point and the sampling head, with \(i\in\left[0,\ N_{point}\right)\), \(j\in\left[0,\ N_{head}\right)\):  
\[\left\{\left(\Delta x_{ij},\Delta y_{ij}\right)\right\}=Linear(Q_{i-1}) \tag{5}\]  
\[P_{ij}=C_{l-1}+\left\{\left(\Delta x_{ij},\Delta y_{ij}\right)\right\} \tag{6}\]  
Subsequently, through bilinear interpolation sampling, we sequentially completed the sampling of multiscale features layer by layer to obtain the sampled feature \(f\in\mathbb{R}^{N_{p}\times N_{C}}\):  
\[f=Bilinear\left(F,\ P_{ij}\right) \tag{7}\]  
where \(F\) represents the multiscale features, \(N_{P}=3\times N_{point}\times N_{head}\) is the number of sampling points, and \(N_{C}=\frac{256}{N_{head}}\) is the channel size.  
AdaMixer [26] is an efficient method for the adaptive decoding of sampled features, termed adaptive mixing. This approach focuses solely on the sampled features, not requiring computation for all features. Following center-guided adaptive sampling, we utilized adaptive mixing to generate a query update based on the sampled features.  
Furthermore, our decoder adopts a structure with cross-attention followed by self-attention. Specifically, the decoder first utilizes adaptive mixing to update queries through sampled features and then further integrates the relationships between queries through self-attention.