# Panoptic Segmentation Method Based on Feature Fusion and Edge Guidance
## 3 Method
### Dual-Path Decoder  
Conventional Transformer decoders predominantly employ single-path update strategies (e.g., Panoptic-DeepLab [7] exclusively updates query embeddings, while MaskFormer [9] emphasizes pixel feature optimization), implementing layer-wise alternating updates of either pixel embeddings or query embeddings. However, this unidirectional information flow may result in inadequate feature interaction, particularly when segmenting dense instances of identical categories, often leading to mask overlap issues. Drawing inspiration from the EM algorithm [14], we propose a dual-path update strategy that models pixel-instance membership relationships as latent variables, approximating the maximum likelihood solution through alternating optimization of pixel features (E-step) and queries (M-step).  
The theoretical framework can be formally expressed as:  
\[\text{argmax}_{\theta}\mathbb{E}_{z\sim p(z|x)}[\log p(x,z|\theta)] \tag{16}\]  
where the alternating optimization of pixel feature x and query parameter \(\theta\) establishes a bidirectional information pathway.  
Pixel Feature Update: Given the current query, the similarity between each pixel feature and individual queries is computed through the cross-attention mechanism. This similarity metric effectively represents the posterior probability of pixel membership to each target instance. Subsequently, the pixel features are updated as the weighted average of the queries, based on these posterior probabilities.  
\[\alpha_{ij}=\frac{\exp(q_{i}^{T}k_{j})}{\sum_{k=1}^{N}\exp(q_{i}^{T}k_{k})} \tag{17}\]  
\[X^{new}=\sum_{i=1}^{M}\alpha_{ij}v_{i} \tag{18}\]  
where q, k, and v represent the query, key, and value vectors, respectively, which are derived through linear projection operations.  
Query Update: Given the current pixel features and their corresponding posterior probabilities, each query is updated through masked attention and self-attention mechanisms. The masked attention mechanism confines the query's attention scope to the foreground region of the mask predicted by the preceding layer, effectively weighting the query based on pixel posterior probabilities. The self-attention mechanism facilitates information exchange among queries, further refining their representations. Through this alternating update paradigm, the dual-path strategy progressively optimizes both pixel features and queries, enabling mutual adaptation and ultimately converging to a local optimal solution. This process bears similarity to the EM algorithm's iterative optimization between latent variables and model parameters, ultimately approximating the maximum likelihood estimation.  
\[\text{MaskAttn}(Q,X)=\text{Softmax}(\frac{QW_{Q}(XW_{K}\odot M)^{T}}{\sqrt{d}}) XW_{V} \tag{19}\]where \(M\in\{0,1\}^{H\times W}\) represents the foreground mask, while \(\odot\) signifies element-wise multiplication.  
Compared to conventional single-path update strategies, the dual-path approach offers several distinct advantages: (1) Enhanced information interaction: While single-path updates typically focus solely on either query-pixel interactions or pixel-level self-attention, the dual-path strategy facilitates more comprehensive information exchange between pixel features and queries through alternating updates, enabling better capture of both global and local relationships within the image. (2) Improved optimization efficiency: The bidirectional optimization inherent in the dual-path strategy accelerates convergence to optimal solutions, thereby reducing the required number of decoder layers. (3) Superior feature representation: The synergistic interaction between pixel features and queries in the dual-path framework generates more refined feature representations, consequently enhancing segmentation accuracy.