## 2 Materials and Methods
### The Location of Objects  
#### 2.3.1 Aligning RGB Image and Depth Image  
Object recognition (color recognition) requires the RGB image, but the RGB image lacks the depth information of objects, which leads to the difficulty of locating an object. Therefore, the depth image is necessary to obtain the objects' depth information. The RGB image with a resolution of 1280 \(\times\) 720, and the resolution of the depth image is 480 \(\times\) 270. The difference in the resolution between the two images is significant, so it is necessary to align the two images, which means finding the corresponding pixels between the depth image and the RGB image. This study can directly calculate the corresponding pixels in the depth image through the pixels in the RGB image. While the traditional alignment method traverses the depth image in advance, calculating the corresponding pixels in the RGB image, it then obtains the corresponding pixels in the depth image through the pixels in the RGB image. The method of this paper uses the RGB image to recognize the objects and determine the bounding box of the objects. The bounding box is presented by four values \((x,y,w,h)\), where \((x,y)\) represents the coordinates of the upper left corner of the bounding box and \((w,h)\) represents the width and height of the bounding box. The coordinates of the bounding box's center \(((x+w)/2,(y+h)/2)\) are used to represent the coordinates of the object, and this value is used to obtain the depth value  
Figure 3: Communication settings between devices. of the object in the depth image. Assuming that the pixel coordinate of the point \(P\) in the depth image is \(\left[ \begin{matrix}\mathrm{d}_{\mathrm{j}}&\mathrm{d}_{\mathrm{i}}&1\end{matrix}  \right]^{\top}\), it is calculated as follows:  
\[ \begin{bmatrix}\mathrm{d}_{\mathrm{j}}\\ \mathrm{d}_{\mathrm{i}}\\ 1\end{bmatrix} =\mathrm{K}_{\mathrm{D}}Z_{\mathrm{D}}^{-1} \begin{bmatrix} \mathrm{X}_{\mathrm{D}}\\ \mathrm{Y}_{\mathrm{D}}\\ \mathrm{Z}_{\mathrm{D}}\end{bmatrix}  \tag{1}\]  
where \(\mathrm{K}_{\mathrm{D}}\) is the intrinsic matrix of the depth camera; \(\mathrm{Z}_{\mathrm{D}}\) is the depth value of \(\mathrm{P}\) in the depth camera coordinate system; and \(\left[\begin{matrix}\mathrm{X}_{\mathrm{D}}&\mathrm{Y}_{\mathrm{D}}&\mathrm{ Z}_{\mathrm{D}}\end{bmatrix}\right]^{\top}\) presents the coordinate of \(\mathrm{P}\) in the depth camera coordinate system. The coordinate points from the RGB camera coordinate system to the depth camera coordinate system can be calculated as follows:  
\[ \begin{bmatrix}\mathrm{X}_{\mathrm{D}}\\ \mathrm{Y}_{\mathrm{D}}\\ \mathrm{Z}_{\mathrm{D}}\end{bmatrix} =\mathrm{R}_{\mathrm{R}}^{\mathrm{D}} \begin{bmatrix}\mathrm{X}_{\mathrm{R}}\\ \mathrm{Y}_{\mathrm{R}}\\ \mathrm{D}_{\mathrm{R}}\end{bmatrix} +\mathrm{T}_{\mathrm{R}}^{\mathrm{D}} \tag{2}\]  
where \(\mathrm{R}_{\mathrm{R}}^{\mathrm{D}}\) presents the rotation matrix from the RGB camera coordinate system to the depth camera coordinate system, and \(\mathrm{T}_{\mathrm{R}}^{\mathrm{D}}\) is the transformation matrix from the RGB camera coordinate system to the depth camera coordinate system. \(\left[\begin{matrix}\mathrm{X}_{\mathrm{R}}&\mathrm{Y}_{\mathrm{R}}&\mathrm{ Z}_{\mathrm{R}}\end{bmatrix}\right]^{\top}\) represents the point coordinates in the RGB camera coordinate system, which can be calculated as follows:  
\[ \begin{bmatrix}\mathrm{X}_{\mathrm{R}}\\ \mathrm{Y}_{\mathrm{R}}\\ \mathrm{Z}_{\mathrm{R}}\end{bmatrix} =\mathrm{K}_{\mathrm{R}}^{-1} \begin{bmatrix} \mathrm{r}_{\mathrm{i}}\\ \mathrm{r}_{\mathrm{i}}\\ 1\end{bmatrix} Z_{\mathrm{R}} \tag{3}\]  
where \(\mathrm{K}_{\mathrm{R}}\) is the intrinsic matrix of the depth camera, \(Z_{\mathrm{R}}\) is the depth value of point \(\mathrm{P}\) in the RGB camera coordinate system, and \(\left[ \begin{matrix}\mathrm{r}_{\mathrm{i}}&\mathrm{r}_{\mathrm{i}}&1\end{matrix}  \right]^{\top}\) presents the coordinate of point \(\mathrm{P}\) in the pixel coordinate system of the RGB camera. The \(\left[