## 4 Human-Computer Interaction Framework
### Uniform Coordinate System  
In order to facilitate subsequent data processing, we need to make uniform the 3D coordinate information of the skeleton nodes captured by the 6 Kinect V2 into one coordinate system, as shown in Figure 4. To avoid noise interference, all captured skeletal node information is preprocessed using median filtering.  
\(C_{i}\) represents the coordinate system centered on the \(i\)th Kinect. We take multiple sets of checkerboard calibration board images with different poses in the overlapping area of the two Kinects, and then use stereo calibration method to calculate the transformation matrix \(H_{i,j}\) between \(C_{i}\) and \(C_{j}\).  
\[\left\{\begin{aligned} & P_{l}=R_{l}\cdot P_{w}+T_{l}\\ & P_{r}=R_{r}\cdot P_{w}+T_{r}\end{aligned}\right., \tag{4}\]  
where \(P_{w}\) represents the coordinates of a point \(P\) on the checkerboard calibration board in the world coordinate system. \(R_{l}\) and \(T_{l}\) represent the rotation matrix and translation matrix of point \(P\) relative to the left Kinect position, respectively. \(R_{r}\) and \(T_{r}\) represent the rotation matrix and translation matrix of point \(P\) relative to the right Kinect position, respectively. \(P_{l}\) and \(P_{r}\), respectively, represent the coordinates of the left and right Kinects in the world coordinate system. With \(P_{w}\), Formula (4) can be rewritten as:  
\[(R_{l})^{-1}\cdot(P_{l}-T_{l})-(R_{r})^{-1}\cdot(P_{r}-T_{r})=0. \tag{5}\]  
Figure 3: Gesture-based human-computer interaction framework.  
Solving Formula (5), we can get:  
\[P_{r}=R\cdot P_{l}+T. \tag{7}\]  
By combining Formulas (6) and (7), we can get:  
\[\left\{\begin{array}{ll}R=R_{r}R_{l}^{T}\\ T=T_{r}-RT_{l}\end{array}\right.. \tag{8}\]  
The transformation matrix \(H\) can be expressed as:  
\[H=\begin{vmatrix}R&T\\ 0&1\end{vmatrix}. \tag{9}\]  
The point \(P_{i,k}\) in the coordinate system \(C_{i}\) can obtain the coordinate \(P_{j,k}\) of this point in the coordinate system \(C_{j}\) by transforming the matrix \(H_{i,j}\), that is \(P_{j,k}=H_{i,j}\ P_{i,k}\). We take \(C_{1}\) as the coordinate system of the interaction area of the whole system. By obtaining the transformation matrix between every two Kinects, we can unify the points in other coordinate systems into the \(C_{1}\) coordinate system, as follows:  
\[\left\{\begin{array}{ll}P_{1,0}=H_{0,1}P_{0,0}\\ P_{1,2}=H_{2,1}P_{2,2}\\ P_{1,3}=H_{3,4}H_{4,1}P_{3,3}\\ P_{1,4}=H_{4,1}P_{4,4}\\ P_{1,5}=H_{5,4}H_{4,1}P_{5,5}\end{array}\right.. \tag{10}\]  
Figure 4: Uniform coordinate system.  
The user's skeletal nodes may be captured by \(n\) Kinects at the same time, where \(n\geq 1\). Then, after the coordinates of the skeletal joint point in each Kinect coordinate system are unified to the \(C_{1}\) coordinate system by Formula (10), the coordinate of the skeletal node \(P\) can be expressed as:  
\[P=\frac{1}{n}\sum_{i=1}^{n}P_{i},i=1,2\ldots n. \tag{11}\]  
The advantage of this processing is that, on the one hand, the stability of the coordinate \(P\) can be improved through multi-coordinate system fusion. On the other hand, when the user is performing interactive operations, if some skeletal nodes are blocked relative to a certain Kinect, the spatial coordinates of the node can be obtained with the help of other Kinects, so the robustness of the interaction can be improved to a certain extent.