# Real-Time Capture of Snowboarder’s Skiing Motion Using a 3D Vision Sensor

Authors: Zhipeng Li; Jun Wang; Tao Zhang; Dave Balne; Bing Li; Ruizhu Yang; Wenli Song; Xingfu Zhang
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Due to the influence of environmental interference and too fast speed, there are some problems in ski motion capture, such as inaccurate motion capture, motion delay, and motion loss, resulting in the inconsistency between the actual motion of later athletes and the motion of virtual characters. To solve the above problems, a real-time skiing motion capture method of snowboarders based on a 3D vision sensor is proposed. This method combines the Time of Fight (TOF) camera and high-speed vision sensor to form a motion acquisition system. The collected motion images are fused to form a complete motion image, and the pose is solved. The pose data is bound with the constructed virtual character model to drive the virtual model to synchronously complete the snowboarding motion and realize the real-time capture of skiing motion. The results show that the motion accuracy of the system is as high as 98.6%, which improves the capture effect, and the motion matching proportion is better and more practical. It is also excellent in the investigation of motion delay and motion loss.


---

## Body

## 1. Introduction

Snowboarding is an intense sport, which is mainly carried out in winter, and it is one of the necessary events of the Winter Olympic Games [1]. The sport is exciting, cool, and dangerous, and the training requirements are rather high.Physical quality consists of technology, psychology, tactics, and other factors. In recent years, the development of physical fitness training in China has made great progress than before. In theory, we no longer blindly follow the traditional training mode of “three importance and one greatness,” and the training means and methods are constantly innovated. However, China’s physical training development is still in the stage of catching up and learning, and there is still a big gap with foreign countries. In foreign countries, physical training pays more attention to the comprehensive and balanced development of physical elements and puts forward many powerful concepts of physical training. Functional physical training is a popular training concept in recent years, but due to the lack of theoretical knowledge of functional physical training in China, we still lack a comprehensive and systematic understanding. This study collects a large number of the latest relevant literature and foreign literature, summarizes, arranges, and applies them to cross-country skiing physical training, so as to make a certain contribution to improving the competitive ability of cross-country skiing teams. Motion capture, also known as motion tracking, collects athletes’ motion data in real time and then uses these data to drive virtual character simulation to display athletes’ motions [2]. In the skiing motion training of professional athletes, motion capture is very important. It can be used to observe athletes’ moving posture in real time, which is of great practical significance to correct wrong motions and improve athletes’ skill level. At present, the accuracy of the motion capture system used by Chinese athletes in skiing training is not high. Because the skiing motion is too fast and difficult, it is easy to miss capture. If there is no more professional system to capture the motion of skiers for a long time, the coach will not be able to correct their mistakes in the skiing process in time, which will seriously affect the training speed of skiers [3]. To solve the above problems, a more suitable method for skiing motion capture is proposed, that is, snowboarding athletes’ skiing motion real-time capture based on a 3D vision sensor. 3D display technology is a new technology with rapid development in recent years. At present, it is widely used in animation production. Compared with the 2D presentation effect, 3D display technology can present the picture more realistically and make people feel immersive. 3D vision sensor is the main device to realize 3D technology. Based on the original system, 3D technology is applied to this study in order to provide effective help for ski training.The main contributions of this paper are as follows:(1)
Reduce missed capture and improve the accuracy of the system in capturing motion(2)
The application of a 3D vision sensor in real-time capture of snowboarding athletes’ skiing motion improves the performance of motion capture, which is convenient to capture feature points and better simulate motion(3)
Different datasets are used to simulate the skiing motion real-time capture system in this paper, and the feasibility of the system is tested


## 2. Related Work

Based on the above background, motion capture technology has been widely used in various sports professional training. For example, in literature [4], Wang et al. established a multiview camera system to capture moving human body images from multiple views in real time, then estimated human body 3D key points from multiview images using the OpenPose method, reconstructed dense 3D point clouds from the images, and finally fitted the SMPL model to 3D key points and 3D point clouds to represent human motion. However, the motion of this method is too complex. In literature [5], Guo and Song proposed a hand running direction capture method based on soft sensing. This method uses HTC Vive equipment to capture human hand motion and then combines the double-layer LSTM to estimate the hand motion and clarify the hand motion characteristics. However, the matching proportion of this method is low. In literature [6], Yu et al. proposed a human motion capture method based on a single RGB-D sensor. This method realizes motion tracking refinement at the body part level through semantic tracking loss calculation, which can improve the tracking accuracy in the case of severe occlusion and fast motion. However, the motion accuracy of this method is low. In literature [7], Maruyama et al. proposed a MoCap capture method based on an inertial measurement unit. This method does not need any optical equipment and can measure the motion attitude even in the outdoor environment. However, the motion delay of this method is too long. Literature [1] proposed to use a series of precise physiological and biochemical instruments to detect the physical function of athletes, so as to master the law of the change of athletes’ mechanical energy in sports training, scientifically organize and arrange the training and practical activities of snowboarding skills on veneer U-shaped venues, and reasonably guide athletes’ diet. This can effectively prevent sports injury, prolong sports life, and improve sports ability. However, due to the loss of motion in this method, it is impossible to accurately capture snowboarding athletes’ skiing motion.Compared with the above systems, this paper uses a TOF camera and high-speed vision sensor as a motion acquisition system. This can greatly improve the accuracy of system motion simulation, and it is less affected by environmental interference. It can be applied to motion capture in various environments and has a good application prospect.


## 3. Real-Time Capture of Snowboarding Motion Using a 3D Vision Sensor

Snowboarding is dangerous, and motion learning is difficult. Ordinary people generally need several months to ensure that they can slide alone. However, for professional athletes with competition pressure, optical gliding is not enough. They also need continuous training to achieve excellent results in the formal competition. The training intensity of professional athletes is much higher than that of ordinary people, especially for the control of some details of motion. In order to assist the training of skiers, a real-time snowboarding capture method based on a 3D vision sensor is proposed in this study. This method mainly solves the following three problems:(1)
Interference by ambient light(2)
The motion speed is too fast to capture(3)
The athlete’s actual motion cannot be consistent with the virtual character model motionThe first two problems are the main reasons for the last problem. Therefore, the latter problem can be solved by solving the first two problems and finally improved the accuracy of motion capture. This study is divided into three parts: snowboarding motion data acquisition, pose solution, and virtual character model establishment and driving. Specific analysis is made for these three parts.


### 3.1. Snowboarding Motion Image Acquisition

In order to solve the influence of ambient light interference and motion speed on motion capture, this design uses two kinds of 3D vision sensors to form an acquisition system to improve the comprehensiveness and integrity of motion data acquisition [8–10]. One of the two sensors is a TOF camera with less interference from ambient light, and the other is a high-speed vision sensor, which can be used to capture fast-moving targets. Through the combination of the two sensors, the problem of data loss in motion capture is reduced.


#### 3.1.1. TOF Depth Camera

TOF depth camera collects reflected light to build depth mapped images. The distance from the camera to the object can be calculated according to time and the speed of light. Formula (1) is as follows:
(1)D=c⋅Δt2,whereD refers to the distance and Δt refers to the time for the pulse signal to travel from the camera and the target and then return [11].


#### 3.1.2. High-Speed Vision Sensor

Compared with ordinary sensors, high-speed vision sensors adopt stacked structure and back-illuminated pixel array to improve their parallel processing ability [12, 13]. According to relevant research, the speed of detecting and tracking targets by high-speed vision sensor can reach 1000 frames per second. In this paper, a high-speed vision sensor is used to collect the data of dynamic target in real time, so as to reduce the missed acquisition probability of motion.After the snowboarding athletes’ skiing motion images are collected by the two 3D vision sensors, image fusion is required to integrate the two images. The fusion process is as follows:Step 1.
Multiscale transformation.The motion images of two kinds of source snowboarding are decomposed by wavelet transform, and two kinds of coefficients are obtained, namely, high-frequency coefficients and low-frequency coefficients.(1)
Edge featureStep 2.
Feature extraction.The Canny operator is used to analyze the low-frequency coefficients of motion images of two kinds of source snowboardingYj,A (low-frequency coefficient of TOF depth image) and Yj,B (low-frequency coefficient of high-speed vision sensor image) for the extraction of edge features and then construct binary edge feature Zj,A and Zj,B [14]. The construction formula is as follows:
(2)Zj,A=XORYj,A,Yj,BANDYj,A,Zj,B=XORYj,A,Yj,BANDYj,B,whereXOR refers to the XOR operation symbol, XORYj,A,Yj,B means to calculate the low-frequency coefficients of TOF depth image and high-speed vision sensor image, and AND means the calculation symbol.
(2)
Gradient featureThe gradient feature extraction formula [15] is as follows:
(3)Si,j=∑i=1X∑j=1YΔfx2i,j2+Δfy2i,j2/2X⋅Y,whereSi,j refers to the average gradient of the pixel at the low-frequency coefficient of i,j, Δfxi,j and Δfyi,j are the difference of the image at the regional window’s central pixel i,j at direction x and y, and X means the number of pixels within the regional window at the direction x and y.
(3)
Extraction of signal intensity featureFor the high-frequency coefficients after image decomposition, the signal intensity features are extracted. The extraction formula is as follows:(4)RAB=Gj,AGj,B,whereRAB means the intensity ratio of correlation signal, Gj,A means signal intensity in TOF depth image area window, and Gj,B means signal intensity in image region window of a high-speed vision sensor.Step 3.
Feature integration.According to the above features, the fusion is carried out, and the fusion formula is as follows:(5)HABi,j=Zj,A⋅Sj,Ai,j⋅RAB+Zj,B⋅Sj,Bi,j⋅RAB,whereHABi,j means the integrated feature of the two kinds of images, Sj,Ai,j means the gradient feature of the TOF depth image, and Sj,Bi,j means the gradient feature of the high-speed visual sensor image [16].Step 4.
Multiscale inverse transform.The fused high- and low-frequency coefficients are transformed by wavelet multiscale inverse transform to obtain the fused athlete motion image.


### 3.2. Snowboarding Motion Pose Solution

Based on the fused image, the pose is solved. The solution process is shown in Figure1. Attitude solution is based on Euler angle.Figure 1
Pose solution process.According to Figure1. The TOF camera image calibrated by the camera and the high-speed vision sensor image are first input into the distortion correction module for image correction, then the image is binarized, the gray value of image pixels is set, and the image with black-and-white effect is obtained. The image data is transmitted to the target recognition module to recognize it, extract the most table and depth value of the target center, then calculate the three-dimensional coordinates and relative distance matrix of the image target center, and finally enter the target matching. When the matching logarithm is greater than 3, the position can be solved. If it is less than 3, the three-dimensional coordinates and relative distance matrix of the image center need to be recalculated. After the image pose is solved, the next image can be solved. Pose solution is the key and core. Pose solution is to calculate the pose data of the target [17, 18]. Suppose the same point f is defined as f in the 3D coordinate of the object coordinate system Oc-XcYcZc, and the 3D coordinate in the camera coordinate system Oo-XoYoZo is defined as g(6)f=Rq+V,whereR represents the rotation matrix and V represents the translation matrix.It is transformed into the following form and then optimized to solve the rotation matrix and translation matrix.(7)f′=∑i=1nwiRfi+V−gi2,wheref′ is the optimized solution, fi is the coordinate value of the target i in Oc-XcYcZc, gi is the coordinate value of the same target i in Oo-XoYoZo, and wi is the weight of the target i in calculation [19].Then calculate(8)f̄=∑i=1nwifi∑i=1nwi,ḡ=∑i=1nwigi∑i=1nwi,wheref̄ is the weight center of the target in Oc-XcYcZc, ḡ is the weight center of the target in Oo-XoYoZo, wi is the weight of target i in calculation, fi is the coordinate value of the target i in Oc-XcYcZc, gi is the coordinate value of the same target i in Oo-XoYoZo, i is the initial value of the summation calculation, i=1, and n is the end value of the summation calculation.Find the center vector of the target in two coordinate systems.(9)αi=fi−f̄,βi=gi−ḡ,wherei=1,2,⋯,n, αi and βi are the central vector of the target in Oc-XcYcZc and Oo-XoYoZo, fi is the coordinate value of the target i in Oc-XcYcZc, f̄ is the weight center of the target in Oc-XcYcZc, gi is the coordinate value of the same target i in Oo-XoYoZo, and ḡ is the weight center of the target in Oo-XoYoZo.This forms the center vector matrixα=α1,α2,⋯,αn and β=β1,β2,⋯,βn. This calculates the scale factor S. The calculation formula is as follows:
(10)S=αWβT,whereW is the diagonal matrix composed of wi and T is the transpose symbol. Decompose the S matrix and then get the rotation matrix R and the translation matrix V.
(11)R=xyT⋅detxyT,V=g¯−Rf¯,wherex,y refers to the pixel coordinate, ḡ refers to the weight center of the target in Oo-XoYoZo, f̄ refers to the weight center of the target in Oc-XcYcZc, R is the rotation matrix, V is the translation matrix, xyT is the rotation matrix of the pixel coordinate x,y, and detxyT is the determinant of the transposed matrix of the pixel coordinate x,y.Finally, the rotation matrixR is transformed into Euler angle form to obtain snowboarding motion data.


### 3.3. Establishment and Driving of the Virtual Character Model

The establishment of the virtual human model is to establish a moving three-dimensional human model. On this basis, combined with the calculated snowboarding motion data, the synchronous driving of virtual character motion can be realized. The specific process is divided into the following three steps:Step 1.
Establish a human skeleton model. Human movement is mainly driven by bones. The human skeleton structure can be simplified into 16 main bones, and all movement postures can be reflected on these 16 bones, as shown in Figure2.Step 2.
Skin the bones. In order to ensure the fidelity of the established athlete virtual model, a layer of human-like skin is covered outside the skeleton to improve the three-dimensional of the model.Step 3.
Fit the snowboarding motion data obtained in the previous chapter with the manikin.Step 4.
Judge whether the fitting curve is close to the motion trajectory. If it is close, the output bone changes continuously; otherwise, refit.Step 5.
Drive the virtual model to complete a series of skiing motions.Figure 2
Human skeleton model.


## 3.1. Snowboarding Motion Image Acquisition

In order to solve the influence of ambient light interference and motion speed on motion capture, this design uses two kinds of 3D vision sensors to form an acquisition system to improve the comprehensiveness and integrity of motion data acquisition [8–10]. One of the two sensors is a TOF camera with less interference from ambient light, and the other is a high-speed vision sensor, which can be used to capture fast-moving targets. Through the combination of the two sensors, the problem of data loss in motion capture is reduced.


### 3.1.1. TOF Depth Camera

TOF depth camera collects reflected light to build depth mapped images. The distance from the camera to the object can be calculated according to time and the speed of light. Formula (1) is as follows:
(1)D=c⋅Δt2,whereD refers to the distance and Δt refers to the time for the pulse signal to travel from the camera and the target and then return [11].


### 3.1.2. High-Speed Vision Sensor

Compared with ordinary sensors, high-speed vision sensors adopt stacked structure and back-illuminated pixel array to improve their parallel processing ability [12, 13]. According to relevant research, the speed of detecting and tracking targets by high-speed vision sensor can reach 1000 frames per second. In this paper, a high-speed vision sensor is used to collect the data of dynamic target in real time, so as to reduce the missed acquisition probability of motion.After the snowboarding athletes’ skiing motion images are collected by the two 3D vision sensors, image fusion is required to integrate the two images. The fusion process is as follows:Step 1.
Multiscale transformation.The motion images of two kinds of source snowboarding are decomposed by wavelet transform, and two kinds of coefficients are obtained, namely, high-frequency coefficients and low-frequency coefficients.(1)
Edge featureStep 2.
Feature extraction.The Canny operator is used to analyze the low-frequency coefficients of motion images of two kinds of source snowboardingYj,A (low-frequency coefficient of TOF depth image) and Yj,B (low-frequency coefficient of high-speed vision sensor image) for the extraction of edge features and then construct binary edge feature Zj,A and Zj,B [14]. The construction formula is as follows:
(2)Zj,A=XORYj,A,Yj,BANDYj,A,Zj,B=XORYj,A,Yj,BANDYj,B,whereXOR refers to the XOR operation symbol, XORYj,A,Yj,B means to calculate the low-frequency coefficients of TOF depth image and high-speed vision sensor image, and AND means the calculation symbol.
(2)
Gradient featureThe gradient feature extraction formula [15] is as follows:
(3)Si,j=∑i=1X∑j=1YΔfx2i,j2+Δfy2i,j2/2X⋅Y,whereSi,j refers to the average gradient of the pixel at the low-frequency coefficient of i,j, Δfxi,j and Δfyi,j are the difference of the image at the regional window’s central pixel i,j at direction x and y, and X means the number of pixels within the regional window at the direction x and y.
(3)
Extraction of signal intensity featureFor the high-frequency coefficients after image decomposition, the signal intensity features are extracted. The extraction formula is as follows:(4)RAB=Gj,AGj,B,whereRAB means the intensity ratio of correlation signal, Gj,A means signal intensity in TOF depth image area window, and Gj,B means signal intensity in image region window of a high-speed vision sensor.Step 3.
Feature integration.According to the above features, the fusion is carried out, and the fusion formula is as follows:(5)HABi,j=Zj,A⋅Sj,Ai,j⋅RAB+Zj,B⋅Sj,Bi,j⋅RAB,whereHABi,j means the integrated feature of the two kinds of images, Sj,Ai,j means the gradient feature of the TOF depth image, and Sj,Bi,j means the gradient feature of the high-speed visual sensor image [16].Step 4.
Multiscale inverse transform.The fused high- and low-frequency coefficients are transformed by wavelet multiscale inverse transform to obtain the fused athlete motion image.


## 3.1.1. TOF Depth Camera

TOF depth camera collects reflected light to build depth mapped images. The distance from the camera to the object can be calculated according to time and the speed of light. Formula (1) is as follows:
(1)D=c⋅Δt2,whereD refers to the distance and Δt refers to the time for the pulse signal to travel from the camera and the target and then return [11].


## 3.1.2. High-Speed Vision Sensor

Compared with ordinary sensors, high-speed vision sensors adopt stacked structure and back-illuminated pixel array to improve their parallel processing ability [12, 13]. According to relevant research, the speed of detecting and tracking targets by high-speed vision sensor can reach 1000 frames per second. In this paper, a high-speed vision sensor is used to collect the data of dynamic target in real time, so as to reduce the missed acquisition probability of motion.After the snowboarding athletes’ skiing motion images are collected by the two 3D vision sensors, image fusion is required to integrate the two images. The fusion process is as follows:Step 1.
Multiscale transformation.The motion images of two kinds of source snowboarding are decomposed by wavelet transform, and two kinds of coefficients are obtained, namely, high-frequency coefficients and low-frequency coefficients.(1)
Edge featureStep 2.
Feature extraction.The Canny operator is used to analyze the low-frequency coefficients of motion images of two kinds of source snowboardingYj,A (low-frequency coefficient of TOF depth image) and Yj,B (low-frequency coefficient of high-speed vision sensor image) for the extraction of edge features and then construct binary edge feature Zj,A and Zj,B [14]. The construction formula is as follows:
(2)Zj,A=XORYj,A,Yj,BANDYj,A,Zj,B=XORYj,A,Yj,BANDYj,B,whereXOR refers to the XOR operation symbol, XORYj,A,Yj,B means to calculate the low-frequency coefficients of TOF depth image and high-speed vision sensor image, and AND means the calculation symbol.
(2)
Gradient featureThe gradient feature extraction formula [15] is as follows:
(3)Si,j=∑i=1X∑j=1YΔfx2i,j2+Δfy2i,j2/2X⋅Y,whereSi,j refers to the average gradient of the pixel at the low-frequency coefficient of i,j, Δfxi,j and Δfyi,j are the difference of the image at the regional window’s central pixel i,j at direction x and y, and X means the number of pixels within the regional window at the direction x and y.
(3)
Extraction of signal intensity featureFor the high-frequency coefficients after image decomposition, the signal intensity features are extracted. The extraction formula is as follows:(4)RAB=Gj,AGj,B,whereRAB means the intensity ratio of correlation signal, Gj,A means signal intensity in TOF depth image area window, and Gj,B means signal intensity in image region window of a high-speed vision sensor.Step 3.
Feature integration.According to the above features, the fusion is carried out, and the fusion formula is as follows:(5)HABi,j=Zj,A⋅Sj,Ai,j⋅RAB+Zj,B⋅Sj,Bi,j⋅RAB,whereHABi,j means the integrated feature of the two kinds of images, Sj,Ai,j means the gradient feature of the TOF depth image, and Sj,Bi,j means the gradient feature of the high-speed visual sensor image [16].Step 4.
Multiscale inverse transform.The fused high- and low-frequency coefficients are transformed by wavelet multiscale inverse transform to obtain the fused athlete motion image.


## 3.2. Snowboarding Motion Pose Solution

Based on the fused image, the pose is solved. The solution process is shown in Figure1. Attitude solution is based on Euler angle.Figure 1
Pose solution process.According to Figure1. The TOF camera image calibrated by the camera and the high-speed vision sensor image are first input into the distortion correction module for image correction, then the image is binarized, the gray value of image pixels is set, and the image with black-and-white effect is obtained. The image data is transmitted to the target recognition module to recognize it, extract the most table and depth value of the target center, then calculate the three-dimensional coordinates and relative distance matrix of the image target center, and finally enter the target matching. When the matching logarithm is greater than 3, the position can be solved. If it is less than 3, the three-dimensional coordinates and relative distance matrix of the image center need to be recalculated. After the image pose is solved, the next image can be solved. Pose solution is the key and core. Pose solution is to calculate the pose data of the target [17, 18]. Suppose the same point f is defined as f in the 3D coordinate of the object coordinate system Oc-XcYcZc, and the 3D coordinate in the camera coordinate system Oo-XoYoZo is defined as g(6)f=Rq+V,whereR represents the rotation matrix and V represents the translation matrix.It is transformed into the following form and then optimized to solve the rotation matrix and translation matrix.(7)f′=∑i=1nwiRfi+V−gi2,wheref′ is the optimized solution, fi is the coordinate value of the target i in Oc-XcYcZc, gi is the coordinate value of the same target i in Oo-XoYoZo, and wi is the weight of the target i in calculation [19].Then calculate(8)f̄=∑i=1nwifi∑i=1nwi,ḡ=∑i=1nwigi∑i=1nwi,wheref̄ is the weight center of the target in Oc-XcYcZc, ḡ is the weight center of the target in Oo-XoYoZo, wi is the weight of target i in calculation, fi is the coordinate value of the target i in Oc-XcYcZc, gi is the coordinate value of the same target i in Oo-XoYoZo, i is the initial value of the summation calculation, i=1, and n is the end value of the summation calculation.Find the center vector of the target in two coordinate systems.(9)αi=fi−f̄,βi=gi−ḡ,wherei=1,2,⋯,n, αi and βi are the central vector of the target in Oc-XcYcZc and Oo-XoYoZo, fi is the coordinate value of the target i in Oc-XcYcZc, f̄ is the weight center of the target in Oc-XcYcZc, gi is the coordinate value of the same target i in Oo-XoYoZo, and ḡ is the weight center of the target in Oo-XoYoZo.This forms the center vector matrixα=α1,α2,⋯,αn and β=β1,β2,⋯,βn. This calculates the scale factor S. The calculation formula is as follows:
(10)S=αWβT,whereW is the diagonal matrix composed of wi and T is the transpose symbol. Decompose the S matrix and then get the rotation matrix R and the translation matrix V.
(11)R=xyT⋅detxyT,V=g¯−Rf¯,wherex,y refers to the pixel coordinate, ḡ refers to the weight center of the target in Oo-XoYoZo, f̄ refers to the weight center of the target in Oc-XcYcZc, R is the rotation matrix, V is the translation matrix, xyT is the rotation matrix of the pixel coordinate x,y, and detxyT is the determinant of the transposed matrix of the pixel coordinate x,y.Finally, the rotation matrixR is transformed into Euler angle form to obtain snowboarding motion data.


## 3.3. Establishment and Driving of the Virtual Character Model

The establishment of the virtual human model is to establish a moving three-dimensional human model. On this basis, combined with the calculated snowboarding motion data, the synchronous driving of virtual character motion can be realized. The specific process is divided into the following three steps:Step 1.
Establish a human skeleton model. Human movement is mainly driven by bones. The human skeleton structure can be simplified into 16 main bones, and all movement postures can be reflected on these 16 bones, as shown in Figure2.Step 2.
Skin the bones. In order to ensure the fidelity of the established athlete virtual model, a layer of human-like skin is covered outside the skeleton to improve the three-dimensional of the model.Step 3.
Fit the snowboarding motion data obtained in the previous chapter with the manikin.Step 4.
Judge whether the fitting curve is close to the motion trajectory. If it is close, the output bone changes continuously; otherwise, refit.Step 5.
Drive the virtual model to complete a series of skiing motions.Figure 2
Human skeleton model.


## 4. Experimental Analysis and Results

### 4.1. Experimental Environment and Dataset

PKU MMD is a large long sequence multimodal and multiview dataset released in 2017. The images in the dataset have the characteristics of multiview, including 1076 untrimmed long sequence albums and 20000 trimmed samples; miniImageNet is a subset of ImageNet dataset, which contains more than 100 categories; each category contains 600 pictures. At present, it is widely used in small sample simulation experiments. In this paper, PKU MMD and miniImageNet datasets are selected as the datasets. The training parameters of snowboarder’s skiing motion are shown in Table1.Table 1
Training parameters of snowboarder’s skiing motion.
ParametersValue (or running state)Number of input images20000 sample imagesIterations800OptimizerRandom gradient descentInitial value of weight correction0.01There are many snowboarding motions, among which the air grabbing is the most typical one. Athletes need to complete technical movements such as turning and somersault with a veneer. This technology has a large range of motion, has high requirements for the mastery of athletes’ skill level, and is also one of the necessary items for athletes’ training. Based on this, taking the skiing movement as the research object, the motion sequence images are collected by the visual acquisition system, the acquisition frequency is 0.20 frames/s, a total of 100 sequence images are collected, and the time is 20 s.The TOF camera in the vision acquisition system is the Intel RealSense l515 radar TOF camera depth realistic camera, which can be used under various indoor and outdoor lighting conditions, and the image resolution can reach1280×720 30 fps, the maximum working range is 0.4~20 m, and the accuracy error is less than 2.7%. At the same time, it is equipped with IMU to realize the automatic monitoring of the moving route of the object. The high-speed vision sensor is imx382, which can detect and track the moving target at the speed of 1000 frames per second, so as to avoid the problem of missing image information due to the moving speed of the target during image capture. The algorithm is used for motion detail comparison, and the process is shown in Figure 3.Figure 3
Comparison method of skiing motion details.The pose solution results are given to the established athlete virtual model to drive the model to move synchronously, and then, the consistency between the athlete’s actual motion and the virtual character model motion is compared.Using the above research method, a series of motion poses in snowboarding are calculated, and the Euler angles of each bone node are obtained, and then, the line diagram is shown in Figure4.Figure 4
Line image segmentation results.
(a)
Standard motion(b)
Virtual character modelThe standard motion line image in Figure4 and the motion line image presented by the virtual character model are segmented at equal time intervals and then analyzed according to the experimental indicators.


### 4.2. Experimental Indicators

There are two main evaluation indexes for skiing motion comparison, and the consistency of motion capture is comprehensively analyzed from these two indexes. These two items are analyzed in detail in the following:(1)
Motion complexityThe more feature points, the more complex the motion is. By comparing the key points between the virtual character model and the standard motion, the execution results of the two groups of skiing motions are judged.(12)X=Mv−CNr,whereMv represents the accuracy of motion extraction and CNr represents the accuracy of motion extraction.
(2)
Matching proportionMatching proportion refers to the ratio between the matching results of two groups of motion key points and the number of standard motion features. The greater the ratio, the higher the matching degree.(3)
Motion accuracy: obtain motion accuracy by comparing the number of frames(4)
Motion delay rate: obtain the motion delay of each system by comparing video animation(5)
Motion loss rate: obtain the loss of motion execution by comparing each image


### 4.3. Results and Discussion

In order to verify the effectiveness of snowboarding athletes’ skiing motion real-time capture based on a 3D vision sensor, experimental analysis was carried out. Under the same test conditions, carry out motion capture according to the four methods given in literature [4] to literature [1], and then carry out fine motion comparison, and compare the comparison results with the studied methods. It is shown in Table 2.Table 2
Comparison results of key points in motion.
CategoryMethodStandard motionThe proposed methodLiterature [4] methodLiterature [5] methodLiterature [6] methodLiterature [7] methodLiterature [1] methodMotion complexityOne segment58585655555654Second segment125126120122120124120Third segment256250236237233235237Fourth segment240238230224220227224Fifth segment1051031001011009896Matching proportionOne segment580.9580.8560.8470.9100.8770.798Second segment1250.9920.8740.8720.9020.8690.850Third segment2560.9250.8850.8650.8780.8920.882Fourth segment2400.9720.9020.8770.8930.8740.865Fifth segment1050.9210.8880.8860.8730.8820.872Motion accuracy (%)One segment100989089878586Second segment100989590868790Third segment100999391858582Fourth segment100999693909797Fifth segment10999596919085Motion delay rate (ms)One segment0155060403020Second segment0144560443521Third segment0124858432828Fourth segment0154957402929Fifth segment0135255392625Motion loss rateOne segmentUnclearUnclearClearClearClearUnclearClearSecond segmentUnclearUnclearClearClearClearClearUnclearThird segmentUnclearUnclearUnclearUnclearClearClearClearFourth segmentUnclearUnclearUnclearClearClearClearUnclearFifth segmentUnclearUnclearClearUnclearClearUnclearClearAccording to Table2, compared with the other five methods, the difference between the key points of 100 frame standard motion sequence images and 100 frame virtual character motion sequence images is the smallest, and the matching proportion with the standard motion sequence image is larger. This shows that the skiing motion displayed by the virtual character is more consistent with the standard motion, which proves that the motion capture data collected by the research method in this paper is more comprehensive. In addition, from the segmentation results, the error mainly occurs in the third and fourth segments. The motion of these two segments is the most complex. During image acquisition, it is easy to miss the acquisition of image information or collect in advance and delay, resulting in deviation from the standard motion. Finally, the presentation of the virtual model is not completely consistent with the standard motion. Compared with other methods, the accuracy of this method is as high as 98.6%, which is much higher than other systems, and its motion delay and loss are also better than other systems. The system studied in this paper has a better capture effect.According to Figure5, the motion matching ratio of literature [4] method is 1.15, the motion matching ratio of literature [5] method is 1.1, the feature extraction stability of literature [6] method is 1.05, the motion matching ratio of literature [7] method is 1, the motion matching ratio of literature [1] method is 0.9, and the motion matching ratio of the method in this paper is 1.2. Therefore, the snowboarding motion matching proportion using the method in this paper is better and more practical.Figure 5
Comparison of matching proportion of skiing motion.According to Figure6, the motion accuracy of literature [4] method is 0.183, the motion accuracy of literature [5] method is 0.178, the motion accuracy of literature [6] method is 0.186, the motion accuracy of literature [7] method is 0.176, the motion accuracy of literature [1] method is 0.186, and the motion accuracy of the method in this paper is 0.195. It can be seen that the snowboarding motion using the method in this paper has high accuracy and improves the capture effect.Figure 6
Comparison of skiing motion accuracy.According to Figure7, the motion delay rate of literature [4] method is 0.12, the motion delay rate of literature [5] method is 0.15, the motion delay rate of literature [6] method is 0.10, the motion delay rate of literature [7] method is 0.13, the motion delay rate of literature [1] method is 0.18, and the motion delay rate of the method in this paper is 0.03. Therefore, the snowboarding motion delay rate using the method in this paper is lower.Figure 7
Comparison of skiing motion delay rate.According to Figure8, the motion loss rate of literature [4] method is 6.8, the motion loss rate of literature [5] method is 7.2, the motion loss rate of literature [6] method is 7.8, the motion loss rate of Literature [7] method is 6, the motion loss rate of literature [1] method is 5.2, and the motion loss rate of the method in this paper is 3.4. Therefore, the snowboarding motion loss rate using the method in this paper is low, and the motion capture is more accurate.Figure 8
Comparison of skiing motion loss rate.


## 4.1. Experimental Environment and Dataset

PKU MMD is a large long sequence multimodal and multiview dataset released in 2017. The images in the dataset have the characteristics of multiview, including 1076 untrimmed long sequence albums and 20000 trimmed samples; miniImageNet is a subset of ImageNet dataset, which contains more than 100 categories; each category contains 600 pictures. At present, it is widely used in small sample simulation experiments. In this paper, PKU MMD and miniImageNet datasets are selected as the datasets. The training parameters of snowboarder’s skiing motion are shown in Table1.Table 1
Training parameters of snowboarder’s skiing motion.
ParametersValue (or running state)Number of input images20000 sample imagesIterations800OptimizerRandom gradient descentInitial value of weight correction0.01There are many snowboarding motions, among which the air grabbing is the most typical one. Athletes need to complete technical movements such as turning and somersault with a veneer. This technology has a large range of motion, has high requirements for the mastery of athletes’ skill level, and is also one of the necessary items for athletes’ training. Based on this, taking the skiing movement as the research object, the motion sequence images are collected by the visual acquisition system, the acquisition frequency is 0.20 frames/s, a total of 100 sequence images are collected, and the time is 20 s.The TOF camera in the vision acquisition system is the Intel RealSense l515 radar TOF camera depth realistic camera, which can be used under various indoor and outdoor lighting conditions, and the image resolution can reach1280×720 30 fps, the maximum working range is 0.4~20 m, and the accuracy error is less than 2.7%. At the same time, it is equipped with IMU to realize the automatic monitoring of the moving route of the object. The high-speed vision sensor is imx382, which can detect and track the moving target at the speed of 1000 frames per second, so as to avoid the problem of missing image information due to the moving speed of the target during image capture. The algorithm is used for motion detail comparison, and the process is shown in Figure 3.Figure 3
Comparison method of skiing motion details.The pose solution results are given to the established athlete virtual model to drive the model to move synchronously, and then, the consistency between the athlete’s actual motion and the virtual character model motion is compared.Using the above research method, a series of motion poses in snowboarding are calculated, and the Euler angles of each bone node are obtained, and then, the line diagram is shown in Figure4.Figure 4
Line image segmentation results.
(a)
Standard motion(b)
Virtual character modelThe standard motion line image in Figure4 and the motion line image presented by the virtual character model are segmented at equal time intervals and then analyzed according to the experimental indicators.


## 4.2. Experimental Indicators

There are two main evaluation indexes for skiing motion comparison, and the consistency of motion capture is comprehensively analyzed from these two indexes. These two items are analyzed in detail in the following:(1)
Motion complexityThe more feature points, the more complex the motion is. By comparing the key points between the virtual character model and the standard motion, the execution results of the two groups of skiing motions are judged.(12)X=Mv−CNr,whereMv represents the accuracy of motion extraction and CNr represents the accuracy of motion extraction.
(2)
Matching proportionMatching proportion refers to the ratio between the matching results of two groups of motion key points and the number of standard motion features. The greater the ratio, the higher the matching degree.(3)
Motion accuracy: obtain motion accuracy by comparing the number of frames(4)
Motion delay rate: obtain the motion delay of each system by comparing video animation(5)
Motion loss rate: obtain the loss of motion execution by comparing each image


## 4.3. Results and Discussion

In order to verify the effectiveness of snowboarding athletes’ skiing motion real-time capture based on a 3D vision sensor, experimental analysis was carried out. Under the same test conditions, carry out motion capture according to the four methods given in literature [4] to literature [1], and then carry out fine motion comparison, and compare the comparison results with the studied methods. It is shown in Table 2.Table 2
Comparison results of key points in motion.
CategoryMethodStandard motionThe proposed methodLiterature [4] methodLiterature [5] methodLiterature [6] methodLiterature [7] methodLiterature [1] methodMotion complexityOne segment58585655555654Second segment125126120122120124120Third segment256250236237233235237Fourth segment240238230224220227224Fifth segment1051031001011009896Matching proportionOne segment580.9580.8560.8470.9100.8770.798Second segment1250.9920.8740.8720.9020.8690.850Third segment2560.9250.8850.8650.8780.8920.882Fourth segment2400.9720.9020.8770.8930.8740.865Fifth segment1050.9210.8880.8860.8730.8820.872Motion accuracy (%)One segment100989089878586Second segment100989590868790Third segment100999391858582Fourth segment100999693909797Fifth segment10999596919085Motion delay rate (ms)One segment0155060403020Second segment0144560443521Third segment0124858432828Fourth segment0154957402929Fifth segment0135255392625Motion loss rateOne segmentUnclearUnclearClearClearClearUnclearClearSecond segmentUnclearUnclearClearClearClearClearUnclearThird segmentUnclearUnclearUnclearUnclearClearClearClearFourth segmentUnclearUnclearUnclearClearClearClearUnclearFifth segmentUnclearUnclearClearUnclearClearUnclearClearAccording to Table2, compared with the other five methods, the difference between the key points of 100 frame standard motion sequence images and 100 frame virtual character motion sequence images is the smallest, and the matching proportion with the standard motion sequence image is larger. This shows that the skiing motion displayed by the virtual character is more consistent with the standard motion, which proves that the motion capture data collected by the research method in this paper is more comprehensive. In addition, from the segmentation results, the error mainly occurs in the third and fourth segments. The motion of these two segments is the most complex. During image acquisition, it is easy to miss the acquisition of image information or collect in advance and delay, resulting in deviation from the standard motion. Finally, the presentation of the virtual model is not completely consistent with the standard motion. Compared with other methods, the accuracy of this method is as high as 98.6%, which is much higher than other systems, and its motion delay and loss are also better than other systems. The system studied in this paper has a better capture effect.According to Figure5, the motion matching ratio of literature [4] method is 1.15, the motion matching ratio of literature [5] method is 1.1, the feature extraction stability of literature [6] method is 1.05, the motion matching ratio of literature [7] method is 1, the motion matching ratio of literature [1] method is 0.9, and the motion matching ratio of the method in this paper is 1.2. Therefore, the snowboarding motion matching proportion using the method in this paper is better and more practical.Figure 5
Comparison of matching proportion of skiing motion.According to Figure6, the motion accuracy of literature [4] method is 0.183, the motion accuracy of literature [5] method is 0.178, the motion accuracy of literature [6] method is 0.186, the motion accuracy of literature [7] method is 0.176, the motion accuracy of literature [1] method is 0.186, and the motion accuracy of the method in this paper is 0.195. It can be seen that the snowboarding motion using the method in this paper has high accuracy and improves the capture effect.Figure 6
Comparison of skiing motion accuracy.According to Figure7, the motion delay rate of literature [4] method is 0.12, the motion delay rate of literature [5] method is 0.15, the motion delay rate of literature [6] method is 0.10, the motion delay rate of literature [7] method is 0.13, the motion delay rate of literature [1] method is 0.18, and the motion delay rate of the method in this paper is 0.03. Therefore, the snowboarding motion delay rate using the method in this paper is lower.Figure 7
Comparison of skiing motion delay rate.According to Figure8, the motion loss rate of literature [4] method is 6.8, the motion loss rate of literature [5] method is 7.2, the motion loss rate of literature [6] method is 7.8, the motion loss rate of Literature [7] method is 6, the motion loss rate of literature [1] method is 5.2, and the motion loss rate of the method in this paper is 3.4. Therefore, the snowboarding motion loss rate using the method in this paper is low, and the motion capture is more accurate.Figure 8
Comparison of skiing motion loss rate.


## 5. Conclusions

Skiing is a standard event in the Winter Olympics. Because its motion is cool and thrilling, skiing attracts many young people to participate. Compared with other sports, skiing is much more difficult and requires higher precision of athletes’ motion during training. Therefore, in order to better assist athletes to complete training, this paper studies the real-time capture method of snowboarding athletes’ skiing motion based on a 3D vision sensor. By combining two kinds of 3D vision sensors, this study makes up for the problem of information loss during image acquisition of the existing system and improves the accuracy of motion real-time capture. In the test process, the accuracy of the system studied in this paper can reach more than 98.6%, which is much higher than other systems and is better than other systems in motion accuracy, motion delay, and loss. However, this study still needs to be improved. The motion capture test only takes a skiing motion as the test object, and the results have certain limitations, which need to be further analyzed and discussed.



---

*Source: 8517771-2021-11-16.xml*