# Grasp Detection Combining Self-Attention with CNN in Complex Scenes
## 3 Problem Definition  
For vision-based grasp detection, a visual sensor captures a multi-channel image that includes the object under consideration, while assuming the existence of multiple workable grasping configurations within the image. In this work, we employ an improved variant of the grasp rectangle representation as proposed in [17]. Specifically, in the case of a parallel-jaw gripper, a grasp rectangle is defined by the position and orientation of the end effector concerning the target object, as well as the quality of the grasp when executed.  
\[\mathrm{G_{R}}=\{\mathrm{p},\sigma_{\mathrm{R}},\mathrm{w_{R}},\mathrm{h_{R}}, \mathrm{q}\} \tag{1}\]  
where \(\mathrm{p}=\{\mathrm{x},\mathrm{y},\mathrm{z}\}\) is the center coordinates, \(\sigma\) represents the clockwise rotation angle around the Z-axis, \((\mathrm{w_{R}},\mathrm{h_{R}})\) is the grasp rectangle width and height in the robot frame where \(\mathrm{w_{R}}\) also means the opening width of the end effector, and \(\mathrm{q}\) is the grasp quality score.  
The grasp representations in an image frame are given by:  
\[\mathrm{G_{i}}=\{\mathrm{x_{i}},\mathrm{y_{i}},\sigma_{\mathrm{i}},\mathrm{w_{ i}},\mathrm{h_{i}},\mathrm{q}\} \tag{2}\]  
where \(\mathrm{x_{i}}\) and \(\mathrm{y_{i}}\) denote the center coordinates of the grasp rectangle in the image frame, \(\sigma_{\mathrm{i}}\) represents the rotation angle in the image frame, \((\mathrm{w_{i}},\mathrm{h_{i}})\) denotes the grasp rectangle width and height in the image frame, and \(\mathrm{q}\) is the same as in Equation (1).  
Different from the 5-dimensional grasp in [4; 6; 13], the grasp quality \(\mathrm{q}\) expresses the probability of a successful grasp, which is similar to the sample confidence in object detection. In detail, for each pixel, a floating number from 0 to 1, corresponding to the pixel position, is found and quantifies a grasping success, with values close to 1 indicating a higher likelihood. \(\sigma_{\mathrm{i}}\) represents the rotation angle during grasping, with a range defined as \([-\frac{\pi}{2},+\frac{\pi}{2}]\).  
To execute a grasp, the map between the image frame and the robot frame needs to be established, as in the following:  
\[\mathrm{G_{R}}=\mathrm{T_{C}^{R}}\Big{(}\mathrm{T_{i}^{C}}(\mathrm{G_{i}}) \Big{)}\]  
where  
\[\mathrm{T_{i}^{C}}=\begin{bmatrix}\frac{1}{7}&0&0\\ 0&\frac{1}{7}&0\\ 0&0&1\end{bmatrix}\mathrm{T_{C}^{R}}=\begin{bmatrix}\mathrm{R}&\mathrm{T}\end{bmatrix} \tag{3}\]  
\(\mathrm{T_{i}^{C}}\) is the transform matrix from image frame to camera frame, which is related to the focal length \(f\) of the camera. \(\mathrm{T_{C}^{R}}\) is the transformation matrix from image frame to robot frame, consisting of a rotation matrix \(\mathrm{R}\in\mathbb{R}^{3\times 3}\) with a translation matrix \(\mathrm{T}\in\mathbb{R}^{3\times 1}\). \(\mathrm{T_{i}^{C}}\) is obtained by camera calibration [18] and \(\mathrm{T_{C}^{R}}\) by hand-eye calibration [19].  
In the implementation, we calculate each pixel point of the grasping rectangle to obtain the position in the robot frame. The specific calculation process is as follows:  
\[\begin{bmatrix}\mathrm{X_{C}}\\ \mathrm{Y_{C}}\\ \mathrm{Z_{C}}\end{bmatrix}=\mathrm{T_{i}^{C}}*\begin{bmatrix}\mathrm{x_{i}} \\ \mathrm{y_{i}}\\ \mathrm{1}\end{bmatrix}*Z_{C}=\begin{bmatrix}\frac{1}{7}&0&0\\ 0&\frac{1}{7}&0\\ 0&0&1\end{bmatrix}*\begin{bmatrix}\mathrm{x_{i}}\\ \mathrm{y_{i}}\\ \mathrm{1}\end{bmatrix}*Z_{C} \tag{4}\] \[\begin{bmatrix}\mathrm{X_{R}}\\ \mathrm{Y_{R}}\\ \mathrm{Z_{R}}\end{bmatrix}=\mathrm{R}*\begin{bmatrix}\mathrm{X_{C}}\\ \mathrm{Y_{C}}\\ \mathrm{Z_{C}}\end{bmatrix}+\mathrm{T}\]  
where \(Z_{C}\) is the depth in camera frame, directly given by the depth camera. \([\mathrm{X_{C}}\,\mathrm{Y_{C}}\,\mathrm{Z_{C}}]\) is the point in camera frame.