# EGSDK-Net: Edge-Guided Stepwise Dual Kernel Update Network for Panoptic Segmentation
## 3 Method
### Real-Time Edge Guidance Module  
Edge detection and panoptic segmentation are closely related. Edges outline the contours of the objects to be detected, while panoptic segmentation not only divides the areas of each object but also categorizes the segmented regions. As a result, they naturally exhibit similar attributes. The current architecture for panoptic segmentation typically trains masks and categories separately. This decoupling is more advantageous for the integration of edge detection and panoptic segmentation tasks. A critical question to ponder is how to achieve reliable edges and appropriately incorporate them into the information flow. There is no intention to use complex structures to obtain edges, as their primary role is to guide panoptic segmentation in focusing on the contours of different targets. Meanwhile, the edges obtained must be accurate, which poses challenges regarding the attention to spatial locations and the training optimization process. Based on this consideration, we designed and proposed a real-time edge guidance module (RTEGM). The structure of the RTEGM is illustrated in Figure 3. The feature map \(R_{2}\) from the lower stages of the backbone serves as the input for the RTEGM. The process of extracting \(R_{2}\) does not involve cross-token interaction, resulting in it being primarily composed of local information with high resolution. This greatly benefits the extraction of edge features. \(R_{2}\) is first passed through a convolutional layer for additional information extraction and transmission. Subsequently, it is fed into the dual-branch structure of the RTEGM. The design of the first branch is aimed at the application of edge features. Inspired by the spatial attention module in CBAM [29], we performed max pooling and average pooling operations along the channel dimension on \(R_{2}\) to generate feature maps \(R_{max}\) and \(R_{avg}\) at different contextual scales. This section of the design aims to make the module parameters focus more on the positions of edge features after backpropagation. \(R_{max}\) and \(R_{avg}\) are then aggregated to produce the edge feature descriptor \(R_{mix}\), which possesses characteristics from both max pooling and average pooling. This process can be formulated as follows:  
\[R_{mix}=Avgpool(R_{2})+MaxPool(R_{2}), \tag{3}\]  
where \(R_{mix}\in R^{B\times\ C\times\ H\times\ W}\), \(B\) is the batch size, \(C\) indicates the number of feature channels and \(H\) and \(W\) are the height and width of \(R_{5}\), respectively. Unlike CBAM [29], which concatenates the pooled results and uses a large \(7\times 7\) convolution kernel to compress the channels and obtain a 2D spatial attention map, our module does not use convolution to filter the edge feature maps. Instead, we directly use additive aggregation to preserve all edge position information, avoiding the loss or insufficiency of contour information that could limit the improvement of panoptic segmentation performance. Subsequently, \(R_{mix}\) is applied as weights to \(R_{5}^{low}\); this process can be formulated as follows:  
\[R_{5}^{low}=R_{5}^{low}\times(1+R_{mix}). \tag{4}\]Since all edge positions are retained in the previous step to avoid potential noise interference, we add \(1\) to \(R_{mix}\) to additionally reference the original target region contour positioning information. It is worth mentioning that we do not normalize \(R_{mix}\) using the sigmoid function as most methods do. Instead, we use \(R_{mix}\) itself as the weight. Since \(R_{mix}\) contains the essential information from all channels of each pixel in two forms, allowing its value to accurately reflect the significance of each point related to edge features. Using the sigmoid function for normalization inevitably reduces the differences in response intensity concerning this significance. We believe this leads to a waste of extracted edge information and ultimately results in suboptimal performance in edge-guided panoptic segmentation.