# EGSDK-Net: Edge-Guided Stepwise Dual Kernel Update Network for Panoptic Segmentation
## 3 Method
### Stepwise Dual Kernel Update Module  
The core idea of K-Net [2] is to divide feature pixels into \(N\) meaningful groups using learnable convolutional kernels. Because there are no restrictions on the content of the groups, K-Net [2] can perform semantic segmentation, instance segmentation, and panoptic segmentation in the same manner. Initially, K-Net [2] uses \(N\) randomly initialized kernels \(K_{0}\) to generate the initial mask prediction from the image features. As depicted in Figure 4a, K-Net [2] initializes this process with two branches: semantic segmentation and instance segmentation. It generates mask predictions \(M_{inst}\) and \(M_{seg}\) using the corresponding convolutional kernels and image features. However, RT-K-Net [5] argues that this dual-branch structure not only contradicts the idea of a unified architecture but also adds extra computational burden to the model. To address this, they propose simplifying the process  
Figure 3: The structure of the RTEGM. by merging the two branches into a single branch to enhance the role of auxiliary semantic loss. The initialization process of RT-K-Net [5] is shown in Figure 4b. First, the RTFormer segmentation head produces a single feature map \(F\). Then, \(F\) is convolved by the directly initialized panoptic kernel \(K_{0}\) to generate the initial mask prediction \(M_{0}\). Since the position for adding the auxiliary semantic loss has been removed, RT-K-Net [5] introduces an auxiliary semantic segmentation head that is used only during training and has the same structure as the main initialization head. We believe that RT-K-Net [5] makes impressive contributions to the mask and kernel initialization, so we use initialization methods that are essentially consistent with it, as shown in Figure 4c. However, in the subsequent kernel update process, RT-K-Net [5] uses a procedure that is identical to that of K-Net [2]. The primary reason for the kernel update is that the initial \(K_{0}\) does not have enough discriminative ability, which results in the generated \(M_{0}\) performing poorly. Typically, the kernel update process consists of several stages. However, the current process still applies a sequential method for kernel transmission, where the kernel information updated in the previous stage serves only as input for the next stage, which undoubtedly leads to a waste of information. To address this, we make adjustments to the kernel update content based on RT-K-Net [5], depicted in Figure 5. Our basic steps are generally consistent with those of K-Net [2] and RT-K-Net [5], primarily including the following three steps:  
(1) Group Feature Assembling: First, the feature map, \(F\), of the i-th stage is element-wise multiplied by the binarized mask prediction \(B_{i-1}\) from the previous stage to form the group feature, as follows:  
\[F^{k}=\sum_{u}^{H}\sum_{v}^{W}B_{i-1}(u,v)*F(u,v), \tag{5}\]  
where \(F^{k}\in\mathbb{R}^{B\times N\times C}\), \(B\) is the batch size, \(N\) represents the number of groups discussed earlier, which also equals the number of convolution kernels, and \(C\) indicates the number of feature channels. (2) Adaptive Feature Update Since the mask \(M_{i-1}\) from the \((i-1)\)-th stage may be inaccurate, and there may be mutual interference from noise among the different groups, element-wise multiplication is first performed between \(F^{k}\) and \(K_{i-1}\) to derive \(F^{E}\). Two gates, \(G^{F}\) and \(G^{K}\), are learned to evaluate the effects of the kernel and group features on the kernel update process, respectively. Following this, \(G^{F}\) and \(G^{K}\) are applied to weigh the group features and the kernel, completing the adaptive kernel update, as follows:  
\[\widetilde{K}=G^{F}*\varphi_{1}\left(F^{k}\right)+G^{K}*\varphi_{2}(K_{i-1}), \tag{6}\]  
where \(\varphi_{1}\) and \(\varphi_{2}\) are fully connected layers with non-shared weights, followed by layer normalization (LN). Unlike K-Net [2] and RT-K-Net [5], \(\widetilde{K}\in\mathbb{R}^{B\times N\times 2C}\).