# A System for Robotic Extraction of Fasteners
## 3 System Overview  
### Camera Relationships  
As an integration project, this work focuses on digitizing and implementing existing concepts to design a process and augment the automated CRS extraction system to achieve desirable and consistent behavior. The discussion in this section uses mathematical concepts described in various robotics modeling texts, such as the Siciliano text [22]. Many of the mathematical concepts presented in this section were applied using multiple software APIs such as the Intel RealSense pyrealsense2 package and RoboDK's robomath package.  
In this work, \(O_{B}\) represents the base coordinate frame for the robot, \(O_{C}\) represents the coordinate frame for the camera system, and \(O_{O}\) represents the target coordinate frame. In addition to these coordinate frames, a world frame, \(O_{W}\), exists in which all other coordinate frames can be defined in absolute coordinates with respect to one another. Note that the world frame exists at an arbitrary and stationary point in the environment differing from the robot base frame. Figure 6 shows the pinhole camera model. This model describes the mathematical relationship between the coordinates of a point in three-dimensional space and its projection onto an image plane.  
This work focuses on inverting the generic pinhole camera model relationship and deprojecting points from an image projection. Deprojection is the act  
Figure 6: Frontal perspective pinhole camera model for coordinate deprojection.  
camera's 2D image perspective and computing their location in 3D space using camera properties, the image coordinates of said points (in pixels), and relative distance measurements.  
\[p_{x}^{C}=\frac{(X_{I}-X_{O})p_{z}^{C}}{\alpha_{x}f} \tag{1}\]  
\[p_{y}^{C}=\frac{(Y_{I}-Y_{O})p_{z}^{C}}{\alpha_{y}f} \tag{2}\]  
Equations (1) and (2) give the x and y coordinates, respectively, of an object in the camera frame, given the focal length of the camera (\(f\)). \(\alpha_{x}\) and \(\alpha_{y}\) represent the pixel-to-spatial sampling unit conversion (between pixels and millimeters) factors based on the camera's intrinsic properties. The size of the image (in pixels) and its relationship to the center point of the image plane are given by the expressions \(X_{I}-X_{O}\) and \(Y_{I}-Y_{O}\) for the x and y coordinates, respectively. Next, we form the 4 \(\times\) 1 augmented target vector given by Equation (3), in which \(p_{x}^{C}\) and \(p_{y}^{C}\) are obtained from deprojection methods, and \(p_{z}^{C}\) is obtained from the camera stereoscopic sensors which define the object distance with respect to the camera frame \(O_{C}\). To obtain the coordinates of the object in the world reference frame, Equation (5) is used to multiply the \(p_{O}^{C}\) vector by the 4 \(\times\) 4 homogeneous transformation matrix given by Equation (4), which describes the position and orientation of the camera frame in the world frame.  
\[P_{O}^{C}=\left[\begin{array}{c}p_{x}^{C}\\ p_{y}^{C}\\ p_{z}^{C}\\ 1\end{array}\right] \tag{3}\]  
\[T_{C}^{W}=\left[\begin{array}{cc}R_{C}^{W}&P_{C}^{W}\\ 0\ 0\ 0\ \ 1\end{array}\right] \tag{4}\]  
\[P_{O}^{W}=T_{C}^{W}P_{O}^{C} \tag{5}\]  
\[P_{O}^{B}=T_{B}^{W-1}T_{C}^{W}P_{O}^{C} \tag{6}\]  
Alternatively, Equation (6) can be used to represent the target \(P_{O}^{C}\) with respect to the base of the robot \(P_{O}^{B}\), where \(T_{B}^{W}\) is the transformation matrix between the base frame and the world frame. This representation of the coordinates is ideal in software and can be used to issue open-loop move commands to an inverse kinematics endpoint.