# Grasp Detection Combining Self-Attention with CNN in Complex Scenes
## 4 Method
### Transformer-Based Feature Extraction  
The standard transformer is strict about the size of the input, taking as input a 1D sequence of token embeddings. Before being fed into the transformer blocks, the input is evenly cut into flattened 2D patches of the image, and then each is resized into the 1D shape through a projection layer, which yields the vector sequences as the tokens for the contiguous transformer blocks. Specifically, a visual image \(\mathrm{I}\in\mathbb{R}^{\mathrm{W}\times\mathrm{H}\times\mathrm{C}}\) is divided into equally sized patches \(\mathrm{X}\in\mathbb{R}^{\mathrm{N}\times(\mathrm{P}^{\mathrm{Z}}\times\mathrm{ C})}\), where \(\mathrm{(W,H)}\) is the resolution of the original image, \(\mathrm{C}\) is the number of channels, \(\mathrm{P}\) represents the size of each patch, and \(\mathrm{N}~{}=~{}\mathrm{H}\times\mathrm{W}/\mathrm{P}^{2}\) represents the number of total patches. The immediately following projection layer flattens and maps these image patches within \(\mathrm{D}\) dimensions, an effective input sequence length \(\mathrm{X}\in\mathbb{R}^{\mathrm{N}\times\mathrm{D}}\) for the transformer.  
The attention mechanism in the transformer is a crucial component that improves the comparison and combination of local and global features. It has the ability to establish interactions across pixels, regardless of their spatial distance. The structure of the transformer block is presented in Figure 3.  
Figure 2: The architecture of our grasping detection mode.  
In particular, we used multiple-head attention (MHA), which does not share the corresponding parameters, provides flexibility on different features, and reduces processing time due to parallel computing. The attention between image tokens is as follows:  
\[\begin{array}{c}\text{Attention}(\text{Q},\text{K},\text{V})=\text{SoftMax} \Big{(}\frac{\text{Q}\text{K}^{\text{T}}}{\sqrt{\text{Q}}}\Big{)}\text{V}\\ \text{Q}=\text{X}\text{W}_{\text{Q}}^{\text{T}},\text{K}=\text{X}\text{W}_{ \text{K}}^{\text{T}},\text{V}=\text{X}\text{W}_{\text{V}}^{\text{T}},\end{array} \tag{5}\]  
where the Q, K, and V vectors are obtained by multiplying the tokens \(\text{X}\in\mathbb{R}^{\text{N}\times\text{D}}\) by the corresponding weight matrix \(\text{W}_{\text{i}}^{\text{T}}\in\mathbb{R}^{\text{D}\times\text{D}}\), \(\text{W}_{\text{i}}^{\text{T}}\) is implemented by the fully connected layer, and d is a scale parameter.  
For grasp detection in complex backgrounds, the transformer is very helpful with visual awareness. As with [14; 15], we rigorously stacked the equal-sized transformer blocks to extract features from images of complex backgrounds. Accounting for the model runtime matching grasp detection in real-life settings, different sets of parameters were carefully designed to create a delicate balance between speed and accuracy without poor imitation; more details are described in the experiments section. The transformer-based feature extraction uses constant latent vector size D through all of its layers and ends up with the token vectors of the same size as the input \(\text{X}\in\mathbb{R}^{\text{N}\times\text{D}}\), with more holistic semantic information, through a forward propagation. The computation steps of the transformer block are represented as follows:  
\[\begin{array}{c}\text{x}_{\text{i}}^{\prime}=\text{LN}(\text{MHA}(\text{x}_ {\text{i}})+\text{x}_{\text{i}})\\ \text{x}_{\text{i}}^{\prime\prime}=\text{LN}(\text{FFN}(\text{x}_{\text{i}}^{ \prime})+\text{x}_{\text{i}}^{\prime})\end{array} \tag{6}\]  
where \(\text{x}_{\text{i}}\) denotes the output from the previous layer. LN refers to layer norm that normalizes each sample rather than a batch, and FFN is a simple fully connected network.