# EGSDK-Net: Edge-Guided Stepwise Dual Kernel Update Network for Panoptic Segmentation
## 1 Introduction  
Panoptic segmentation is an important task in computer vision, with many real-world applications, such as autonomous driving, relying on the detailed information it provides. Researchers introduced panoptic segmentation [1] based on semantic segmentation and instance segmentation to achieve a more comprehensive understanding of scenes. The latter two categorize the pixels in the image into \"stuff\" and \"thing\" classes, while panoptic segmentation combines the characteristics of both. It requires classifying countable objects in the \"thing\" classes and classifying amorphous regions in the \"stuff\" classes. To better accomplish this task, recent state-of-the-art approaches [2, 3, 4, 5, 6, 7] have employed a unified mask classification method for panoptic segmentation. Among them, RT-K-Net [5] has improved the structure, training, and inference process of K-Net [2], achieving real-time high-performance panoptic segmentation. Nonetheless, we must point out that there are still some clear defects in the existing methods that remain unresolved. First, panoptic segmentation, as a novel task, is still confined within the framework of traditional segmentation tasks. It can be approximately understood as the classification of all relevant objects in the image, but viewing this task exclusively through the perspectives of \"stuff\" and \"thing\" categories has notable limitations. For instance, YOSO [6] introducesa real-time panoptic segmentation framework. By employing dynamic convolutions and feature pyramid aggregation, YOSO achieves competitive performance while reducing computational overhead, making it suitable for real-time applications. Panoptic SwiftNet [7] utilizes scale-equivariant feature extraction, cross-scale upsampling through pyramidal fusion, and boundary-aware learning of pixel-to-instance assignment to achieve fast inference over large input resolutions. Although YOSO [6] and Panoptic SwiftNet [7] demonstrate excellent efficiency and results, they are still limited to the traditional segmentation framework that classifies solely based on the \"stuff\" and \"things\" categories. Moreover, some recent segmentation algorithms [8; 9] have attempted to incorporate boundary information to improve segmentation performance. Ref. [8] introduced an improved Mask R-CNN algorithm for segmenting photovoltaic hot spots in thermal infrared images. The proposed method enhances the edge features of hot spots using residual neural networks and integrates a feature pyramid structure with an edge-guided approach, improving segmentation accuracy significantly. On the other hand, ref. [9] focused on the fusion of multimodal information in high-resolution satellite and airborne remote sensing images. The authors proposed a multimodal fusion network with edge detection guidance to bridge the semantic gap and improve feature extraction. The method enhances segmentation performance by incorporating spatial boundary information and designing an adaptive fusion block to optimize multi-level feature integration. These two works underscore the importance of boundary information in improving segmentation accuracy. However, existing panoptic segmentation methods [2; 3; 4; 5; 6; 7; 10; 11; 12; 13] have limited utilization of boundary information. Hence, we believe that integrating insights from external domains to address the limitations of the traditional framework could be a promising direction for future research. In addition, regarding the kernel initialization of K-Net [2], it employs two branches for mask and kernel initialization. However, these two branches are not essential, as they introduce unnecessary complexity to the model and deviate from the original design concept of a unified structure. Existing real-time panoptic segmentation methods [10; 11; 12; 13] rely on a dual-branch structure for semantic segmentation and instance segmentation, which results in shortcomings in both accuracy and inference speed. Therefore, RT-K-Net [5] takes it a step further by using a single unified branch for initialization.