# Real-Time Interaction for 3D Pixel Human in Virtual Environment
## 3 Pixel-Style Avatar Generation Method
### Pixel Human Visualization
#### 3.3.3 Interactive Arm Generation  
Our study aimed to improve traditional 2D video conferencing applications, where a fixed monocular RGB webcam typically only shows the upper body. Therefore, in this study, we only convert the arms into 3D from an upper-body video stream and realize the function of avatar arm movement according to the actual movements of users. While generating 3D interactive arms, depth maps by real-time depth estimation methods such as [36; 37] are not accurate enough to represent the body details, which affects the results while converting the arms into 3D. The high-precision depth map predicted by [38; 39; 40] can more accurately represents the 3D position and shape of human arms. It can also assist pose estimation in obtaining a more accurate arm range. However, these high-performance depth estimations are highly time-consuming and often used for processing pictures or non-real-time videos rather than real-time applications, such as ours. Human matting and pose estimation are necessary for our real-time pixel avatar generation method. Additional depth estimation method will lead to a sharp drop in the speed of generating avatars, affecting the interactive experience. Therefore, in this work, we use the results of 3D pose estimation to convert the arms of the pixel avatar into 3D according to the 3D skeleton rather than using depth estimation. When determining the arm and hand range that must be converted to 3D, we use the starting and ending points of the upper arms, lower arms, and hands in the virtual space. As shown in Figure 6, the starting point of the upper arm is the shoulder joint, the ending point is the elbow joint. The starting point of the lower arm is the elbow joint, and the ending point is the wrist joint. The starting point of the hand is the wrist joint and extends along the index joint. Figure 5: (**a**) Downsampling-only body point cloud. (**b**) Result of subtracting the edge detection results from the alpha channel of (**a**). (**c**) Histogram equalization on the alpha channel of (**b**). Figure 6: The upper arm range is represented by the orange box from the shoulder joint to the elbow joint and extending to both sides. The lower arm range is represented by the green box from the elbow joint to the index joint and extending to both sides. The hand range is represented by the blue box extending from the wrist joint along the index joint to the first pixel with an alpha value of zero. Because the coordinates of the 3D pose landmarks obtained by pose estimation have different coordinate systems and units compared to those of the avatar joints in the virtual space, we must obtain suitable coordinate transformation parameters. When generating a 2D point cloud avatar, we already obtain the x- and y-axis coordinates of the joints in virtual space, so we only need to know the transformation parameters of the z-axis coordinates. Because the 3D world coordinates of MediaPipe are all in meters, we can calculate the scaling parameters \(z_{scale}\) between the z-axis coordinates in virtual space and z-axis coordinates in world coordinates according to the scaling parameter of the x- and y-axis coordinates. The equation for \(z_{scale}\) is as follows:  
\[z_{scale}=\frac{\left|x_{left\_shoulder}-x_{right\_shoulder}\right|}{\left|X_{ left\_shoulder}-X_{right\_shoulder}\right|} \tag{3}\]  
where \(x_{left\_shoulder}\) and \(x_{right\_shoulder}\) are the coordinates of the left and right shoulders in the virtual space, and \(X_{left\_shoulder}\) and \(X_{right\_shoulder}\) are the world coordinates of the left and right shoulders, respectively. By scaling the z-axis coordinates of the real-world pose landmarks according to \(z_{scale}\), we can obtain virtual space world coordinates of the same scale in all directions. For each part of the arm, we start from the 3D virtual space position of the starting point to perform 3D steps in the direction of the end position and form a one-pixel-wide skeleton from the starting point to the ending point.