# Grasp Detection Combining Self-Attention with CNN in Complex Scenes
## 5 Experiments
### Datasets and Implementation Details  
The Cornell dataset [5], consisting of 885 images, each containing one grasping object from 24 different object categories, has been widely used for grasp detection. Due to the small size of the original dataset, we performed data augmentation, including random cropping, scaling, and rotation, to meet training requirements. Additionally, we conducted experiments on a bigger and more complex multi-object dataset, the VMRD [21], to validate the model. The VMRD, consisting of 5185 images containing 17,688 object instances from 31 object categories, provides 51,530 manipulation relationship labels in total. Some examples of the datasets are in Figure 6. In the experiment, all ground truth grasping boxes were used to involve training.  
Two-Dimensional Gaussian-Based Grasp Augmentation: The dataset images were cropped, scaled, and normalized to match the input of the model. For the grasping quality ground truth, previous research [7] filled the grasp rectangle pixels with 0 and 1, assigning  
Figure 6: Instances in the datasets.  
equal probabilities to the object edge and center. In [22], a Gaussian kernel was used to encode the grasp representation and highlight the object center. Furthermore, we adopted and expanded by using a 2D Gaussian kernel to adjust the grasp label at the pixel level in the grasp quality ground truth \(\mathrm{I_{q}}\in\mathbb{R}^{224\times 224}\). This method not only identifies the object's center position but also provides some orientation, as shown in Figure 7.  
Specifically defined as:  
\[\mathrm{Q(P_{i})}=\frac{1}{\left(2\pi\right)^{\mathrm{d}/2}\left|\Sigma\right|^ {1/2}}\exp\left[-\frac{1}{2}(\mathrm{P_{i}}-\mathrm{u_{q}})^{\mathrm{T}} \Sigma_{\mathrm{I_{q}}}^{-1}\left(\mathrm{P_{i}}-\mathrm{u_{q}}\right)\right] \tag{9}\]  
where  
\[\mathrm{P_{i}}=(\mathrm{x_{i}}\,\mathrm{y_{i}})\ \in\mathrm{I_{q}}\]  
where \(\mathrm{d}=2\) represents the dimension of the vector; \(\mathrm{u_{q}}=\left\{\mathrm{u_{x}}\,\mathrm{u_{y}}\right\}\) represents \((\mathrm{x_{i}}\,\mathrm{y_{i}})\) of the mean; \(\Sigma\) is the covariance matrix, describing the correlation of \((\mathrm{x_{i}}\,\mathrm{y_{i}})\).  
\[\Sigma=\begin{pmatrix}\delta_{1}&\delta_{2}\\ \delta_{3}&\delta_{4}\end{pmatrix} \tag{10}\]  
The covariance matrix \(\Sigma\) is set empirically to conform to the orientation of objects in the data set as much as possible.  
Training configuration: The entire grasp system was achieved using Pytorch 1.11.0 with CUDA 11.3 packages on Ubuntu 20.0. During the training period, the model was trained end-to-end on an AMD Ryzen 5 5600X CPU and an Nvidia RTX3080 Ti GPU with 12 GB of memory.  
Training schedule: We used a three-stage training strategy for the models. First, the grasping CNN of the model was trained on target datasets alone; the best-performing weights were used for the next stage of training. Second, we froze the CNN section and trained the rest sections including the transformer block and the FBB. Third, finally, we unfroze the grasp CNN and trained the entire model end-to-end. During the training process, the stochastic gradient descent (SGD) optimizer was used to optimize the model's backpropagation, with an initial learning rate of 0.001 and a learning rate decay of 0.1.  
For the Cornell dataset, the batch size was set to 16, and the model was trained for a total of 200 epochs, with 50 epochs for the first stage, 50 epochs for the second stage, and 100 epochs for the final stage. For the VMRD, the batch size was set to 8, and the model was trained for a total of 500 epochs, which were split into 100, 150, and 250 epochs, respectively, for each stage. In each training phase, we periodically saved the weight of the model and tested it, and proceeded to the next phase of training when the performance had leveled off.  
Figure 7: The grasp quality label map with 2D Gaussian kernel. Subfigure (**a**) is the original image, and subfigure (**b**) is its grasp quality label.