# Panoptic Segmentation Method Based on Feature Fusion and Edge Guidance
## 3 Method  
### Network Infrastructure Overview  
The proposed Panoptic Segmentation Model based on Feature Fusion and Edge Guidance (PSM-FFEG) network architecture, illustrated schematically in Figure 1, follows a pipeline designed to enhance multi-scale feature representation and boundary accuracy. It leverages initial features from a standard backbone and processes them through dedicated encoding, guidance, and decoding stages.  
Backbone Network: We employ ResNet50 as the feature extractor. It provides multi-level feature maps C1, C2, C3, C4 from its stages 1 through 4, with spatial resolutions progressively reduced by factors of 4, 8, 16, 32 relative to the input image size. These features serve as the foundational input for subsequent modules. Notably, the high-resolution C1 feature map is crucial input for the Edge Guidance Layer due to its retention of fine spatial details.  
Feature Encoding Layer: This layer aims to generate powerful multi-scale semantic representations. It consists of:  
* Dynamic Large Kernel (DLK) Modules: Applied sequentially to the backbone features C1, C2, C3, C4 with intermediate downsampling, producing initial multi-scale features E1, E2, E3, E4 with enhanced receptive fields.
* Dynamic Feature Fusion (DFF) Modules: These modules perform adaptive cross-scale fusion between adjacent features from the DLK modules (e.g., fusing E1 and E2, E2 and E3, etc.) to produce refined features E1', E2', E3'. The deepest feature E4 (renamed E4' for consistency notation later) doesn't undergo DFF fusion at this stage but provides deep semantic context. The output E1', E2', E3', E4' captures richer context than the original backbone features.  
Edge Guidance Layer: This layer focuses specifically on refining boundary information.
* Edge Guidance Fusion (EGF) Module: It takes the shallow, high-resolution C1 feature from the backbone and the deep \(E_{4}^{\prime}\) feature from the encoder. It fuses them to generate an edge-enhanced feature map, \(E_{edge}^{\prime}\) guided by an auxiliary edge loss during training.
* Multi-scale Edge Injection: The learned edge feature \(E_{edge}^{\prime}\) is then downsampled to match the resolutions of \(\{E_{1}^{\prime},E_{2}^{\prime},E_{3}^{\prime}\}\) and integrated with them using a gating mechanism. This process yields the final multi-scale feature pyramid \(\{F_{1},F_{2},F_{3},F_{4}\}\). These features \(\{F_{1},F_{2},F_{3},F_{4}\}\), spanning resolutions \(1/4\) to \(1/32\), serve as the input to the decoder.  
Dual-Path Decoding Layer: A parallel interactive architecture is utilized to process the feature pyramid. The query path updates instance-aware features through a Transformer decoder, while the pixel path extracts fine-grained details via a convolutional network.  
Figure 1: The overall architecture of the proposed PSM-FFEG network. Input images are processed by a ResNet-50 backbone. The Feature Encoding Layer uses Dynamic Large Kernel (DLK) and Dynamic Feature Fusion (DFF) modules for multi-scale context. The Edge Guidance Layer fuses shallow and deep (E4’) features, applies edge supervision (via EGF module), and injects edge information into the feature pyramid F1–F4. The Dual-Path Decoder (Right side of dotted line) interactively refines pixel embeddings and object queries using F1–F4 to produce final panoptic segmentation outputs (class and mask predictions).  
The dual-path features interact through a cross-attention mechanism, ultimately generating the semantic categories and instance masks necessary for panoptic segmentation tasks. The decoder consists of multiple layers, each refining the features through self-attention and cross-attention mechanisms. The final output includes both semantic segmentation maps and instance masks.