# Refined Prior Guided Category-Level 6D Pose Estimation and Its Application on Robotic Grasping
## 3 Materials and Methods
### Network Structure
#### 3.2.2 Prior Guided Observation Network  
The structure of the Deformation Refine Module is shown in Figure 4. We propose this module inspired by SGPA [28]. Our goal is to introduce the semantic features of the category prior through the Prior Guided Observation Network, providing additional information for the subsequent deformation process. Unlike SGPA, we inject the category prior into the instance point cloud, guiding the transformation of semantic features by generating a heatmap from the transformed point cloud and the original point cloud. The heatmap can be considered a guide for how the instance features change after incorporating the prior information. This same guidance can also be applied to direct image transformations. First, we pass \(F_{p}^{p}\) through the Dimension Alignment Module, which consists of a two-sided MLP (Multi-layer Perceptron), aimed at reducing the dimensionality of \(F_{p}^{p}\) to facilitate its subsequent matching with \(F_{o}^{p}\). Figure 4: Illustration of the framework of the Prior Guided Observation Network. Figure 3: The figure illustrates the overall structure of the model proposed in this paper. The RGB image of the scene is processed through Mask R-CNN to crop the target object’s RGB image. Using this patch, the corresponding depth image of the target object can also be obtained from the scene’s depth image, leading to the extraction of the target object’s point cloud. The target object’s RGB image, point cloud, and category prior are then input into the Prior Guided Observation Network. The outputs are fed into the Deformation Refine Module and the Correspondence Refine Module, respectively, yielding the deformed point cloud and dense correspondence matrix. Through these computations, the NOCS coordinates of the target object are ultimately obtained. Next, we input \(F_{p}^{p}\) and \(F_{o}^{p}\) into the CrossAttention (CA) module, where \(F_{p}^{p}\) serves as the query, and \(F_{o}^{p}\) as the key and value. The computation in this module is performed as follows:  
\[\mathbf{Q}=\mathbf{F}_{p}^{p}\mathbf{W}^{q},\quad\mathbf{K}=\mathbf{F}_{o}^{p} \mathbf{W}^{k},\quad\mathbf{V}=\mathbf{F}_{o}^{p}\mathbf{W}^{v}, \tag{5}\]  
\[F=\text{softmax}((\mathbf{Q}\mathbf{K}^{\top})\mathbf{V}). \tag{6}\]  
In this context, \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) represent the query, key, and value, respectively. \(W^{q}\), \(W^{k}\), and \(W^{v}\) are learnable projection matrices for the query, key, and value, respectively. We utilize a total of \(M\) attention blocks, implementing a multi-head attention mechanism, to thoroughly model the information in \(F_{p}^{p}\) and fully inject it into \(F_{o}^{p}\), facilitating the extraction of more informative semantic features from the category prior. The output features from these \(M\) attention blocks are concatenated as follows:  
\[F_{d}=\text{Concat}(F^{(1)},F^{(2)},\dots,F^{(M)}), \tag{7}\]  
Next, we input \(F_{d}\) into a Feed Forward Network (FFN) to obtain the features \(F_{o}^{d}\) of the instance point cloud, transformed by the prior point cloud. To measure the degree of change in \(F_{o}^{d}\) relative to \(F_{o}^{p}\), we compute a heatmap (\(\mathbf{H}\)) using the cosine similarity between \(F_{o}^{d}\) and \(F_{o}^{p}\):  
\[\mathbf{H}=\textit{cosine}\quad similarity=\frac{\mathbf{A}\cdot\mathbf{B}}{| \mathbf{A}||\mathbf{B}|}, \tag{8}\]  
\[|\mathbf{A}|=\sqrt{\sum_{i=1}^{n}A_{i}^{2}},\quad|\mathbf{B}|=\sqrt{\sum_{i=1 }^{n}B_{i}^{2}}, \tag{9}\]  
where \(\mathbf{A}\) and \(\mathbf{B}\) are vectors that represent \(F_{o}^{d}\) and \(F_{o}^{p}\), respectively. Subsequently, the corresponding category prior semantic feature \(F_{p}^{s}\) can be generated by  
\[(F_{p}^{s})=\text{softmax}(\mathbf{H})\times F_{rgb}, \tag{10}\]  
Next, we concatenate \(F_{o}^{s}\) and \(F_{o}^{p}\) along the channel dimension, and then pass them through the instance local network, which consists of three layers of MLP, to obtain the local features of the observed object, denoted as \(F_{o}^{l}\).