# Design of an Eye-in-Hand Smart Gripper for Visual and Mechanical Adaptation in Grasping
## 5 Visual Adaptivity by Eye-in-Hand System
### Motion Planning
#### 5.3.1 Conversion of Image Coordinates  
To locate the target position, intrinsic and extrinsic matrices are utilized to convert the image coordinates to world coordinates, so that the real-world distance corresponding to each pixel can be determined. This conversion is illustrated in Figure 25. The conversion between the camera image coordinates \(\left[ \begin{array}{cccc}u&v&w\end{array} \right]^{T}\) and global coordinates \(\left[ \begin{array}{cccc}X&Y&Z\end{array} \right]^{T}\) can be expressed as:  
\[\left[ \begin{array}{cccc}u\\ v\\ w\end{array} \right]\ =\ K[R\mid t]\left[ \begin{array}{cccc}X\\ Y\\ Z\\ 1\end{array} \right]\ =\ \left[ \begin{array}{cccc}\alpha&s&u_{0}\\ 0&\beta&v_{0}\\ 0&0&1\end{array} \right]\left[ \begin{array}{cccc}r_{11}&r_{12}&r_{13}&t_{x} \\ r_{21}&r_{22}&r_{23}&t_{y}\\ r_{31}&r_{32}&r_{33}&t_{z}\end{array} \right]\left[ \begin{array}{cccc}X\\ Y\\ Z\\ 1\end{array} \right] \tag{16}\]  
where the multiplication of \(K\) and \([R\mid t]\) forms a projection matrix. This projection matrix can be split into two matrices: intrinsic parameter matrix \(K\) and extrinsic parameter matrix \([R\mid t]\). In Equation (16), \(u_{0}\) and \(v_{0}\) represent the pixels of the image plane center corresponding to the \(X\)- and \(Y\)-axes. Ideally, if the camera has a resolution of 800 \(\times\) 600 pixels, \(u_{0}\) is 400 pixels, and \(v_{0}\) is 300 pixels. The \(\alpha\), \(\beta\), and \(s\) are the calibration parameters in the intrinsic matrices of the camera, where \(\alpha\) and \(\beta\) are the focal lengths of the camera in the \(X\)- and \(Y\)-axes, respectively. Finally, the \(s\) represents the distortion correction factor if the \(X\)- and \(Y\)-axes are not perpendicular to each other. By contrast, the extrinsic parameter matrix \([R\mid t]\) is the transformation between the world coordinate system and the camera coordinate system, which is the transformation matrix \(M_{B}^{C}\) that converts the base frame into a camera frame. Figure 24: Parameters d, t, z, and \(z^{\prime}\): (**a**) The distance d from tool frame to camera frame, (**b**) The distance t from tool frame to object frame, and (**c**) The distance z of camera frame to object frame, and the distance z\({}^{\prime}\) from expectation grabbing position to object frame. There is no available intrinsic matrix information on the Raspberry PI3's 2-D camera data sheet. To determine the intrinsic parameters of the camera that are not commercially available or offered by the Raspberry PI3's 2-D camera used in this study, we adopted the method established by [PERSON] [21] to determine the intrinsic matrix parameters by using a checkerboard to constrain intersection points on the same plane. Using this method, depth information \(Z\) can be ignored in the projection matrix, therefore, Equation (16) can be rewritten as follows:  
\[\left[ \begin{array}{ccc}u\\ v\\ 1\end{array} \right]\ =\ \left[ \begin{array}{ccc}\alpha&s&u_{0}\\ 0&\beta&v_{0}\\ 0&0&1\end{array} \right]\left[ \begin{array}{ccc}r_{11}&r_{12}&t_{x}\\ r_{21}&r_{22}&t_{y}\\ r_{31}&r_{32}&t_{z}\end{array} \right]\left[ \begin{array}{ccc}X\\ Y\\ 1\end{array} \right]\ =\ \left[ \begin{array}{ccc}h_{1}&h_{2}&h_{3}\\ 1\end{array} \right]\left[ \begin{array}{ccc}X\\ Y\\ 1\end{array} \right]\ =\ H\left[ \begin{array}{c}X\\ Y\\ 1\end{array} \right] \tag{17}\]  
Here, the \(H\) matrix is a 3-by-3 matrix derived by multiplying the intrinsic matrix with the modified extrinsic matrix. Using the checkerboard, as suggested by [PERSON] [21], the intrinsic parameters can be determined by decomposing the \(H\) matrix. As the modified extrinsic matrix is not composed of the rotation matrix and translation vector, it cannot be directly decomposed using QR decomposition. To decompose the \(H\) matrix, the two constraints \(r_{1}\) and \(r_{2}\) possess orthogonal properties in the form:  
\[\left[ \begin{array}{ccc}h_{1}&h_{2}&h_{3}\\ \end{array} \right]\ =\ K\left[ \begin{array}{ccc}r_{1}&r_{2}&t\\ \end{array}