# Intelligent Sports Training System Based on Artificial Intelligence and Big Data

Authors: Chunguang Li; Jianbiao Cui
Journal: Mobile Information Systems
 (2021)

---

## Abstract

All activities in training fields are for the improvement of athletes’ competitive abilities. A sports training system is an organizational system to achieve common goals. Competitive ability is one of the main manifestations of the evolution of the training system. With the rapid development of computer technology, people have begun to combine virtual reality and other technologies to achieve scientific sports-assisted training to eliminate traditional sports training that relied purely on experience. Pose estimation obtains the position, angle, and additional information about the human body in the image in a two-dimensional plane or three-dimensional space by establishing the mapping relationship between the human body features and the human body posture. This article demonstrates a golf-assisted training system to realize the transformation from an experience-based sports training method to a human motion analysis method, using artificial intelligence and big data. The swing posture parameters of the trainer and the coach are obtained using the posture estimation of a human body. Based on this information, an auxiliary training system is built. The two parameters of the joint angle trajectory and the posture similarity are used as auxiliary indicators to compare the trainers. The joint angle trajectory is analyzed, and the coach is guided based on the similarity of the posture.


---

## Body

## 1. Introduction

Modern training theory believes that the training process of athletes is a systematic project, which is a process of multidisciplinary integration of the “best” combination of factors. Therefore, to grasp the athlete’s competitive career process [1], it is necessary to clearly understand the athlete’s growth process and the conditions under which the athlete’s training system [2–4] forms and evolves. It is one of the essential prerequisites for the athlete’s multiyear systematic training to become “scientific” [5–7].The diversification and professional development of today’s competitive sports have made sports training [8, 9] a unique human activity, increasingly rich and complex. With the development of sports training practice and the deepening of people’s understanding of the connotation and extension of sports training, more and more people have gradually realized that modern sports training is a process of specialized training for athletes, which continuously improves their performance. It is a process in which competitive ability is its primary objective [10, 11]. For an athlete to engage in competitive training to reach the peak of individual competitive training through systematic training, it is a phased, continuous, and complete training process. This training process usually lasts more than ten years or even more. Then, in this perennial system training, to improve the athletic performance of athletes at a particular stage, not only many people are involved but also many disciplines are involved. The cooperation and operation of these factors constitute the athlete training system. Among them, personnel include coaches, scientific research personnel, management personnel, and service personnel. The personnel regulate athletes’ sleeping style, diet and wearable kits, training methods, and training frequency. They may be involved in the design and construction of facilities on which athletes complete training [12]. The competition in modern competitive sports is becoming increasingly fierce, and athletes’ success often depends on “details.” It requires different personnel to be careful in their areas of responsibility to ensure the smooth improvement of athletes’ competitive abilities so that multidisciplinary knowledge can be fully penetrated into the sports training process, forming a complex system with multiple factors. A multidisciplinary system is an organizational system that achieves common goals, and any such system has certain organizational goals and organizational management problems. It aims to make the athletes’ competitive abilities develop to the ideal state [13].In traditional sports training [14, 15], training methods based on visual observation are usually adopted. With the development of computer vision [1, 2], people begin to use cameras to capture and analyze the athletes’ movements. In sports training, the movements of athletes are analyzed and tracked from the video image sequence. A comparison of the athletes’ postures is made through scientific quantitative analysis of athletes’ movement characteristics and then combines human physiology and physics principles to propose improvements in sports movements. The method assists athletes in training to get rid of traditional sports training purely relying on experience, realize intuitive exercise analysis and guidance, and scientifically improve the level and performance of athletes.Human-centered technology [16–18] and product design have always been the focus of scientific research personnel and scientific research institutions. The application of computer vision technology in sports training has just started. This article is dedicated to applying artificial intelligence [19–21] and big data technology in the field of sports training. Various training suggestions are provided to detect and track the moving human body in the video and analyze sports posture. We use the human motion analysis based on computer vision. Based on the analysis of this paper, it can be seen that the research of auxiliary sports training systems has substantial theoretical research value and practical engineering significance and has broad application prospects. Following are the main contributions of this paper:(1)
This paper uses the Kinect sensor plug-in OpenNI code to automatically extract the three-dimensional data of 15 joint points of the human body. Finally, the manually extracted posture data are compared with the posture data obtained based on the Kinect three-dimensional sensor to complete its accuracy verification.(2)
This paper proposes a background modeling algorithm based on ViBe modeling, which uses neighborhood pixels to create a background model. Experiments show that this method overcomes the shortcomings of traditional methods and can achieve better detection results for dynamic backgrounds.(3)
Based on the model-based human pose estimation algorithm and the model-free pose estimation algorithm, this paper proposes a human pose estimation method based on contour features combined with image processing. The edge detection is performed on the binary contour map obtained by target detection, and the horizontal scanning is performed. Image processing methods such as human body length ratio constraint have realized the posture estimation of human joints. Experiments show that this method can more accurately extract the main joint data of the human body.This paper is organized as follows. The related work is discussed in Section2. The proposed methodology is discussed in Section 3. The experimental results are elaborated in Section 4. Finally, we conclude the paper and provide future research direction in Section 5.


## 2. Related Work

In this section, we discuss the related work pertaining to our approach. First, we discuss the concept of the athlete training system followed by the impact of competition on athletes’ training system, auxiliary training system, and human body pose estimation. We believe the comprehensive review will provide a solid foundation for our proposed work.


### 2.1. Concept of Athlete Training System

There are three main differences between the athlete training system and the conventional training system. First, the athlete training system is part of the entire training system, and in terms of resources, environment, and other factors, the two are the same to some extent. Still, the factors involved in the athlete training system are part of the entire training system. However, not all these factors are contained in the training system during the training of athletes. The second is that the existence of the athlete training system is based on the presence of the athlete’s competitive ability. For example, consider the case when the athlete quits a competitive sports career [22]. In that case, the training system surrounding the athlete will no longer exist, and the training system always exists because of competitive sports. Finally, the athlete training system is a part of the entire competitive sports training system. The athlete training system highlights the athlete’s prominent position in training and takes the athlete’s competitive ability as the state parameter of the evolution of the training system [23].


### 2.2. The Impact of Competition on Athletes’ Training System

In the athlete training system, all training activities are organized around improving athletes’ competitive ability. All athletes participate in competitions to create ideal sports performance. Competitive ability is the ability that athletes must possess to participate in competitive contests. Having a competition inevitably requires athletes to have specific competitive abilities. Only in this way can they show the concept of higher, faster, and stronger competitive sports. Therefore, the emergence and development of the athlete training system must be closely related to the competition and continue to evolve and sort out with the competition level changes. The competition plays a guiding role in the development of the athlete’s training system [24].


### 2.3. Auxiliary Training System

As golf has become more popular and valued by people, many auxiliary training devices related to golf have appeared in recent years. With the application of computer technology in sports training, a group of companies dedicated to the research of assisted golf training systems has emerged worldwide and has achieved significant results. The main methods currently used are the image method, graph analysis method, and portable sensor method. The imaging method captures the athletes’ movements during the swing through image acquisition equipment and then analyzes and evaluates the movements based on past experience. The video method used in the assisted golf training system is the use of golf motion analysis software “MotionCoach” developed by the Canadian software development company Media Vention. The golf swing analysis system was researched by the American sports software development company Gary Brooks and the DartGoffer developed by the Swiss software development company DartFish [25]. MotionCoach processes video manual calibration and combines the training opinions of golf experts in the calibration and classification assessment process and provides more scientific and meaningful training guidance for the trainees.The graph analysis method reprocesses the graph information. It extracts the physical parameters based on collecting the image information of the athlete’s swing process and compares them with the standard template parameters to realize the digitization of classification evaluation. The graph analysis method used in the golf-assisted training system mainly includes the golf-assisted swing motion analyzer developed by the Korean sports software development company GOLFZON, the golf motion analysis and teaching system MATT-T developed by the North American company TaylorMade, and the auxiliary developed by the Canadian software company Focaltron. The MATT-T teaching system uses a high-speed camera to capture the physical parameters of each joint point during the athlete’s swing and uses a computer to draw the physical parameters of these joint points, analyze the data with analysis software, and give the athlete reasonable suggestions.The portable sensor method is a particular sensor device carried by the athlete. The device captures key posture data in real-time during the swing and combines the data analysis algorithm to give scientific guidance. Zepp Technology Co. launched the world’s first portable golf action-assisted training system, Ltd. called GolfSense. The company’s combination of advanced image capture equipment and data classification algorithms enables GolfSense to obtain key swing data of several golf swings accurately. It can also provide trainees with more accurate results based on its robust database and scientific motion analysis. It fully understands your swing and gets intuitive visual feedback and training suggestions with the most reference value.


### 2.4. Human Body Pose Estimation

Singh et al. [26] proposed a solution to estimate the two-dimensional frontal pose of the upper body. This method is mainly aimed at the human body with only the upper body visible, such as standing among flowers, sitting on a sofa, and being blocked by a yacht. This method can also be used to extend to the estimation of the whole body posture. Based on the literature mentioned above, Ibrar et al. optimized the detection process of various parts of the human body and simplified some initialization requirements by adding some constraining conditions [27]. The skin color of the hands is similar to the skin color of the face. The legs are identical to wearing pants and so on, thereby improving the speed of human body posture estimation and the algorithm’s accuracy. Andriluka used the existing single-frame image two-dimensional posture estimation as a transition to estimate the three-dimensional human posture in the sequence image. They established a human joint tree model in a single frame of the image and estimated the posture and position of the joint tree model based on the structural model of the image. Finally, based on the Bayesian maximum posterior probability, the two-dimensional posture information in the joint tree model was evaluated. A back-projection operation was used to estimate the three-dimensional posture data. Daubney used a method based on the image structure model to restore the three-dimensional posture of the human body. They established the mapping relationship between the human body features in the image and the three-dimensional space. According to this mapping relationship, the posture of the human body can be estimated directly in the three-dimensional space without the two-dimensional posture estimation. Sinh et al. used the segmentation plus pose matching method to realize the local pose estimation of the hand and leg limbs in the image.


## 2.1. Concept of Athlete Training System

There are three main differences between the athlete training system and the conventional training system. First, the athlete training system is part of the entire training system, and in terms of resources, environment, and other factors, the two are the same to some extent. Still, the factors involved in the athlete training system are part of the entire training system. However, not all these factors are contained in the training system during the training of athletes. The second is that the existence of the athlete training system is based on the presence of the athlete’s competitive ability. For example, consider the case when the athlete quits a competitive sports career [22]. In that case, the training system surrounding the athlete will no longer exist, and the training system always exists because of competitive sports. Finally, the athlete training system is a part of the entire competitive sports training system. The athlete training system highlights the athlete’s prominent position in training and takes the athlete’s competitive ability as the state parameter of the evolution of the training system [23].


## 2.2. The Impact of Competition on Athletes’ Training System

In the athlete training system, all training activities are organized around improving athletes’ competitive ability. All athletes participate in competitions to create ideal sports performance. Competitive ability is the ability that athletes must possess to participate in competitive contests. Having a competition inevitably requires athletes to have specific competitive abilities. Only in this way can they show the concept of higher, faster, and stronger competitive sports. Therefore, the emergence and development of the athlete training system must be closely related to the competition and continue to evolve and sort out with the competition level changes. The competition plays a guiding role in the development of the athlete’s training system [24].


## 2.3. Auxiliary Training System

As golf has become more popular and valued by people, many auxiliary training devices related to golf have appeared in recent years. With the application of computer technology in sports training, a group of companies dedicated to the research of assisted golf training systems has emerged worldwide and has achieved significant results. The main methods currently used are the image method, graph analysis method, and portable sensor method. The imaging method captures the athletes’ movements during the swing through image acquisition equipment and then analyzes and evaluates the movements based on past experience. The video method used in the assisted golf training system is the use of golf motion analysis software “MotionCoach” developed by the Canadian software development company Media Vention. The golf swing analysis system was researched by the American sports software development company Gary Brooks and the DartGoffer developed by the Swiss software development company DartFish [25]. MotionCoach processes video manual calibration and combines the training opinions of golf experts in the calibration and classification assessment process and provides more scientific and meaningful training guidance for the trainees.The graph analysis method reprocesses the graph information. It extracts the physical parameters based on collecting the image information of the athlete’s swing process and compares them with the standard template parameters to realize the digitization of classification evaluation. The graph analysis method used in the golf-assisted training system mainly includes the golf-assisted swing motion analyzer developed by the Korean sports software development company GOLFZON, the golf motion analysis and teaching system MATT-T developed by the North American company TaylorMade, and the auxiliary developed by the Canadian software company Focaltron. The MATT-T teaching system uses a high-speed camera to capture the physical parameters of each joint point during the athlete’s swing and uses a computer to draw the physical parameters of these joint points, analyze the data with analysis software, and give the athlete reasonable suggestions.The portable sensor method is a particular sensor device carried by the athlete. The device captures key posture data in real-time during the swing and combines the data analysis algorithm to give scientific guidance. Zepp Technology Co. launched the world’s first portable golf action-assisted training system, Ltd. called GolfSense. The company’s combination of advanced image capture equipment and data classification algorithms enables GolfSense to obtain key swing data of several golf swings accurately. It can also provide trainees with more accurate results based on its robust database and scientific motion analysis. It fully understands your swing and gets intuitive visual feedback and training suggestions with the most reference value.


## 2.4. Human Body Pose Estimation

Singh et al. [26] proposed a solution to estimate the two-dimensional frontal pose of the upper body. This method is mainly aimed at the human body with only the upper body visible, such as standing among flowers, sitting on a sofa, and being blocked by a yacht. This method can also be used to extend to the estimation of the whole body posture. Based on the literature mentioned above, Ibrar et al. optimized the detection process of various parts of the human body and simplified some initialization requirements by adding some constraining conditions [27]. The skin color of the hands is similar to the skin color of the face. The legs are identical to wearing pants and so on, thereby improving the speed of human body posture estimation and the algorithm’s accuracy. Andriluka used the existing single-frame image two-dimensional posture estimation as a transition to estimate the three-dimensional human posture in the sequence image. They established a human joint tree model in a single frame of the image and estimated the posture and position of the joint tree model based on the structural model of the image. Finally, based on the Bayesian maximum posterior probability, the two-dimensional posture information in the joint tree model was evaluated. A back-projection operation was used to estimate the three-dimensional posture data. Daubney used a method based on the image structure model to restore the three-dimensional posture of the human body. They established the mapping relationship between the human body features in the image and the three-dimensional space. According to this mapping relationship, the posture of the human body can be estimated directly in the three-dimensional space without the two-dimensional posture estimation. Sinh et al. used the segmentation plus pose matching method to realize the local pose estimation of the hand and leg limbs in the image.


## 3. Methodology

In this section, we discuss our proposed methodology. First, a brief overview is provided about the image acquisition and data extraction from the posture followed by our algorithm that estimates the human body poses. Finally, we discuss the auxiliary training system that relies on our proposed algorithm.


### 3.1. Image Acquisition and Posture Data Extraction

#### 3.1.1. Kinect Sensor for Image Acquisition

In this section, Kinect three-dimensional sensor coding is used to realize image acquisition and data extraction of 15 joint points of the human body. The development environment is Win10x86 + VS2018 + Kinect + OpenNI 1.5.2 + OpenCV 4.2.4, and the driver of Kinect is written on this platform. When used in Xbox software, the Kinect sensor has an actual distance range of 1.2 meters to 3.5 meters. Kinect can capture images with a plane range of 6 per square meter, a horizontal field of view of 57°, and a vertical field of view of 43°. Kinect can capture 30 frames of data per second, and a swing is completed in about 2 seconds. The color image data and depth data can be obtained by calling the ColorMageSteam function and the DepthMageStream function, respectively, and the real-time image acquisition interface generated is shown in Figure1.Figure 1
Trainer swing process diagram.
(a)(b)(c)(d)(e)(f)


#### 3.1.2. Kinect Sensor for Joint Data Capture

To obtain the human skeleton in the OpenNI environment, it is necessary to use the UserGenerator generator in the program. This function detects the departure or appearance of the character by registering the two callback functions of NewUser and LostUser. Once the character appears in the above two situations, the state of the person’s departure or appearance is detected and the callback functions will be called. Two callback functions such as NewUser and LostUser correspond to the appearance and disappearance of characters, respectively. SkeletonCapability is an ability of the generator UserGenerator, which can be used to store character skeleton information. CalibrationStar and CalibrationEnd are used as the two callback functions in SkeletonCapability in the calibration work of obtaining the human skeleton, and they are used at the beginning and end of the calibration, respectively. The coordinate information obtained through the above steps is based on the Kinect camera coordinate system, and the coordinate projection needs to be converted to the screen coordinate system by the ControlRealWorldToProjective method so that it can be displayed more intuitively. The coordinate data converted by the ConvertRealWordToProjective0 method are stored in the skeletonPointsOut array.In order to verify the accuracy of the Kinect sensor, two sets of data are used for comparison. One group is based on the horizontal and vertical coordinates of the 15 joint points captured by the Kinect sensor. One frame is used for analysis, as shown in Figure2. Based on the human joint points captured by Kinect, the picture on the right is based on the manually marked joint points.Figure 2
Joint point map based on (a) Kinect sensor and (b) manual marking.
(a)(b)


### 3.2. Human Body Pose Estimation Algorithm

#### 3.2.1. Model-Based Human Pose Estimation

The model-based human pose estimation method is related to the constructed human model and the prior knowledge of related motions; that is, the system state at each moment is determined by the system state at the previous moment, and the projection of the human body model and the characteristics of each part of the human body in the image are measured. Between the likelihoods, find the optimal matching method to obtain the human body pose parameters. The process of model-based pose estimation methods is generally “prediction, matching, and update”; that is, first establish a human body model (skeleton model, tree diagram model, dynamic model, and so on) and then pass the previous frame of the human body based on the knowledge of the dynamic model. The posture estimation result assumes the posture of the human body in the current frame. Since the initial frame does not have the prediction of the previous frame, the posture needs to be initialized first and finally the postulated human posture result is projected into the actual image plane, and the posture evaluation function is used as the result of revising the hypothesis. The following will specifically introduce the three important components of the model-based method: building a human body model, selecting an observation matching function, and designing a prediction model.The establishment of a human body model is based on the preparation stage of the model. The human body model established, based on prior knowledge, can effectively constrain the feature change space, thereby improving the calculation efficiency and reducing the search range for matching. Human body structure models are divided into one-dimensional, two-dimensional, and three-dimensional models. In the process of human body modeling, commonly used models include skeleton models, silhouette models, three-dimensional skeleton models, and graphic structure models. The skeleton model is the simplest way to represent the structure of the human body. It is mainly composed of points and line segments, representing joint points and bones, respectively, as shown in Figure3. The skeleton model is mainly composed of points and lines, representing the joint points and bones of the human body, respectively. It is the simplest representation method of the human body structure, as shown in Figure 3(a).Figure 3
Human structure model diagram.
(a)(b)(c)In the model-based human body pose estimation method, the observation process is to project a three-dimensional human body model onto the image plane and find the optimal human body poses (parameters) by measuring the likelihood between the projected model and the real image. Commonly used image features in the observation process include optical flow, color, texture, edge, and contour.In the model-based human pose estimation method, the purpose of the prediction process is mainly to reduce the feasible and search space of the pose. Model-based human pose estimation is a process of searching and solving in a space containing a large number of poses to obtain the pose that best matches the current image pose. However, since the three-dimensional pose of the human body is mostly a high-dimensional continuous variable, what is the current challenge? Dense searching, sampling, and solving are realized in the high-dimensional pose space. Many researchers use Gaussian processes to build dynamic models based on the theory that human motion is a continuous change process with time-series relationships and finally realize predictive search in a low-dimensional space. The specific process is to use low-dimensional subspaces to represent a large number of constraints in the process of human movement and use various dimensionality reduction methods to achieve the purpose of reducing the dimensionality of the posture vector.


#### 3.2.2. Human Pose Estimation Based on Contour Edge Features

The use of model-based human pose estimation methods must search in high-dimensional space. The optimization speed is slow, and it is difficult to achieve real-time processing. It has strong adaptability to the model-free human pose estimation methods based on statistical learning, and the results are accurate. However, the algorithm is more complicated, and the time cost of modeling and the labor cost of training samples are relatively high. Based on the matching model-free human body pose estimation method, it is greatly affected by the capacity of the sample set, and the selection of image features is relatively dependent, and the accuracy is not too high, so in combination with the subject of golf sports objects, this paper proposes a human body pose estimation algorithm based on edge contour combined with image processing. Through this algorithm, the joint points of the human body in the picture can be automatically labeled, and the posture data can be obtained. The schematic diagram is shown in Figure4.Figure 4
Schematic diagram of attitude estimation based on the image processing method.Firstly, the Canny edge detection algorithm is used to obtain its contour edge, and then rough image processing methods such as horizontal line scanning and human body length ratio constraints are used so that the coordinates of the human body joint points are obtained to achieve the purpose of posture estimation. The specific content includes the following parts. First, Gaussian filter is used to smooth the image. Image Gaussian filtering can be implemented in two ways: two one-dimensional Gaussian kernels are weighted twice, respectively, and one two-dimensional Gaussian kernel is convolved once. The equation of the one-dimensional Gaussian kernel function is as follows:(1)K=12πσex∗x/2σ∗σ.The parameters are determined to get a one-dimensional kernel vector. The equation of the two-dimensional Gaussian kernel function is as follows:(2)K=12πσ∗σex∗x+y∗y/2σ∗σ.The parameters are determined to get a two-dimensional kernel vector. Next, the two matrices of the partial derivatives in thex and y directions of the image can be calculated by the finite difference of the first-order partial derivative, and the first-order difference can be used to approximate the gradient of the image gray value. The equation of the image gray value gradient (including amplitude and direction) is shown as follows:(3)H1=−1−111,H2=1−11−1,ϕ1m,n=fm,n∗H1x,y,ϕ2m,n=fm,n∗H2x,y,ϕm,n=ϕ12m,n+ϕ22m,n,θϕ=tan−1ϕ2m,nϕ1m,n.The convolution operator used in the Canny algorithm is relatively simple. For example, in the expression,H1 and H2 and ϕ and θ represent the magnitude and direction of the edge gradient, respectively.The position of the edge cannot be completely determined based on the global gradient alone, and nonmaximum suppression of the gradient amplitude is needed to highlight the true edge. This process is an important step for edge detection. The principle of nonmaximum suppression is described as follows (as shown in Figure5).Figure 5
Schematic diagram of nonmaximum suppression.The gradient direction of pointC is shown by the blue line, and its maximum local gray scale is on this line. The intersection of the eight-valued field of point C and the gradient direction is d1 and d2. Comparing the gray value of point C in its eighth whether there is a local maximum in the value field, it can be judged whether point C is an edge point. If the gray value of point C is less than any one of d1 and d2, it means that point C is not a local maximum and point C is not an edge point. Set its gray value to 0.Finally, the double threshold method can reduce the number of false edges. Choose two thresholds: a high threshold and a low threshold. According to the two thresholds, two threshold images are obtained, respectively. The high threshold image obtains an edge image with few false edges, and then the low threshold image is used to connect the high threshold processed edge images into contours so that all edge gaps are closed.


### 3.3. Auxiliary Training System Based on Human Body Pose Estimation Algorithm

Traditional sports training often relies on manual experience guidance and unable to give accurate numerical analysis of key movements. This article will build an auxiliary training system based on the method of human posture estimation for golf. The system takes the swing big data of the trainer and the coach as input and uses the five joint angle trajectory comparison graphs and the posture similarity of the trainer and the coach as input. Auxiliary training indicators are combined with artificial intelligence technology [3, 4] to output posture prediction results. The frame diagram of the auxiliary sports training system built in this paper is shown in Figure 6.Figure 6
Auxiliary training system frame.Auxiliary sports training system function takes the big data image sequence of the trainer and the coach’s swing as input and takes the 5 joint angle trajectory comparison chart of the trainer and the coach and the posture similarity change chart of each frame as the auxiliary training index, compares the difference between the joint angles of the trainer and the coach, and at the same time uses artificial intelligence technology to modify the trainer’s movements according to the similarity of the posture so as to achieve the purpose of analyzing the sports posture and providing training suggestions.


## 3.1. Image Acquisition and Posture Data Extraction

### 3.1.1. Kinect Sensor for Image Acquisition

In this section, Kinect three-dimensional sensor coding is used to realize image acquisition and data extraction of 15 joint points of the human body. The development environment is Win10x86 + VS2018 + Kinect + OpenNI 1.5.2 + OpenCV 4.2.4, and the driver of Kinect is written on this platform. When used in Xbox software, the Kinect sensor has an actual distance range of 1.2 meters to 3.5 meters. Kinect can capture images with a plane range of 6 per square meter, a horizontal field of view of 57°, and a vertical field of view of 43°. Kinect can capture 30 frames of data per second, and a swing is completed in about 2 seconds. The color image data and depth data can be obtained by calling the ColorMageSteam function and the DepthMageStream function, respectively, and the real-time image acquisition interface generated is shown in Figure1.Figure 1
Trainer swing process diagram.
(a)(b)(c)(d)(e)(f)


### 3.1.2. Kinect Sensor for Joint Data Capture

To obtain the human skeleton in the OpenNI environment, it is necessary to use the UserGenerator generator in the program. This function detects the departure or appearance of the character by registering the two callback functions of NewUser and LostUser. Once the character appears in the above two situations, the state of the person’s departure or appearance is detected and the callback functions will be called. Two callback functions such as NewUser and LostUser correspond to the appearance and disappearance of characters, respectively. SkeletonCapability is an ability of the generator UserGenerator, which can be used to store character skeleton information. CalibrationStar and CalibrationEnd are used as the two callback functions in SkeletonCapability in the calibration work of obtaining the human skeleton, and they are used at the beginning and end of the calibration, respectively. The coordinate information obtained through the above steps is based on the Kinect camera coordinate system, and the coordinate projection needs to be converted to the screen coordinate system by the ControlRealWorldToProjective method so that it can be displayed more intuitively. The coordinate data converted by the ConvertRealWordToProjective0 method are stored in the skeletonPointsOut array.In order to verify the accuracy of the Kinect sensor, two sets of data are used for comparison. One group is based on the horizontal and vertical coordinates of the 15 joint points captured by the Kinect sensor. One frame is used for analysis, as shown in Figure2. Based on the human joint points captured by Kinect, the picture on the right is based on the manually marked joint points.Figure 2
Joint point map based on (a) Kinect sensor and (b) manual marking.
(a)(b)


## 3.1.1. Kinect Sensor for Image Acquisition

In this section, Kinect three-dimensional sensor coding is used to realize image acquisition and data extraction of 15 joint points of the human body. The development environment is Win10x86 + VS2018 + Kinect + OpenNI 1.5.2 + OpenCV 4.2.4, and the driver of Kinect is written on this platform. When used in Xbox software, the Kinect sensor has an actual distance range of 1.2 meters to 3.5 meters. Kinect can capture images with a plane range of 6 per square meter, a horizontal field of view of 57°, and a vertical field of view of 43°. Kinect can capture 30 frames of data per second, and a swing is completed in about 2 seconds. The color image data and depth data can be obtained by calling the ColorMageSteam function and the DepthMageStream function, respectively, and the real-time image acquisition interface generated is shown in Figure1.Figure 1
Trainer swing process diagram.
(a)(b)(c)(d)(e)(f)


## 3.1.2. Kinect Sensor for Joint Data Capture

To obtain the human skeleton in the OpenNI environment, it is necessary to use the UserGenerator generator in the program. This function detects the departure or appearance of the character by registering the two callback functions of NewUser and LostUser. Once the character appears in the above two situations, the state of the person’s departure or appearance is detected and the callback functions will be called. Two callback functions such as NewUser and LostUser correspond to the appearance and disappearance of characters, respectively. SkeletonCapability is an ability of the generator UserGenerator, which can be used to store character skeleton information. CalibrationStar and CalibrationEnd are used as the two callback functions in SkeletonCapability in the calibration work of obtaining the human skeleton, and they are used at the beginning and end of the calibration, respectively. The coordinate information obtained through the above steps is based on the Kinect camera coordinate system, and the coordinate projection needs to be converted to the screen coordinate system by the ControlRealWorldToProjective method so that it can be displayed more intuitively. The coordinate data converted by the ConvertRealWordToProjective0 method are stored in the skeletonPointsOut array.In order to verify the accuracy of the Kinect sensor, two sets of data are used for comparison. One group is based on the horizontal and vertical coordinates of the 15 joint points captured by the Kinect sensor. One frame is used for analysis, as shown in Figure2. Based on the human joint points captured by Kinect, the picture on the right is based on the manually marked joint points.Figure 2
Joint point map based on (a) Kinect sensor and (b) manual marking.
(a)(b)


## 3.2. Human Body Pose Estimation Algorithm

### 3.2.1. Model-Based Human Pose Estimation

The model-based human pose estimation method is related to the constructed human model and the prior knowledge of related motions; that is, the system state at each moment is determined by the system state at the previous moment, and the projection of the human body model and the characteristics of each part of the human body in the image are measured. Between the likelihoods, find the optimal matching method to obtain the human body pose parameters. The process of model-based pose estimation methods is generally “prediction, matching, and update”; that is, first establish a human body model (skeleton model, tree diagram model, dynamic model, and so on) and then pass the previous frame of the human body based on the knowledge of the dynamic model. The posture estimation result assumes the posture of the human body in the current frame. Since the initial frame does not have the prediction of the previous frame, the posture needs to be initialized first and finally the postulated human posture result is projected into the actual image plane, and the posture evaluation function is used as the result of revising the hypothesis. The following will specifically introduce the three important components of the model-based method: building a human body model, selecting an observation matching function, and designing a prediction model.The establishment of a human body model is based on the preparation stage of the model. The human body model established, based on prior knowledge, can effectively constrain the feature change space, thereby improving the calculation efficiency and reducing the search range for matching. Human body structure models are divided into one-dimensional, two-dimensional, and three-dimensional models. In the process of human body modeling, commonly used models include skeleton models, silhouette models, three-dimensional skeleton models, and graphic structure models. The skeleton model is the simplest way to represent the structure of the human body. It is mainly composed of points and line segments, representing joint points and bones, respectively, as shown in Figure3. The skeleton model is mainly composed of points and lines, representing the joint points and bones of the human body, respectively. It is the simplest representation method of the human body structure, as shown in Figure 3(a).Figure 3
Human structure model diagram.
(a)(b)(c)In the model-based human body pose estimation method, the observation process is to project a three-dimensional human body model onto the image plane and find the optimal human body poses (parameters) by measuring the likelihood between the projected model and the real image. Commonly used image features in the observation process include optical flow, color, texture, edge, and contour.In the model-based human pose estimation method, the purpose of the prediction process is mainly to reduce the feasible and search space of the pose. Model-based human pose estimation is a process of searching and solving in a space containing a large number of poses to obtain the pose that best matches the current image pose. However, since the three-dimensional pose of the human body is mostly a high-dimensional continuous variable, what is the current challenge? Dense searching, sampling, and solving are realized in the high-dimensional pose space. Many researchers use Gaussian processes to build dynamic models based on the theory that human motion is a continuous change process with time-series relationships and finally realize predictive search in a low-dimensional space. The specific process is to use low-dimensional subspaces to represent a large number of constraints in the process of human movement and use various dimensionality reduction methods to achieve the purpose of reducing the dimensionality of the posture vector.


### 3.2.2. Human Pose Estimation Based on Contour Edge Features

The use of model-based human pose estimation methods must search in high-dimensional space. The optimization speed is slow, and it is difficult to achieve real-time processing. It has strong adaptability to the model-free human pose estimation methods based on statistical learning, and the results are accurate. However, the algorithm is more complicated, and the time cost of modeling and the labor cost of training samples are relatively high. Based on the matching model-free human body pose estimation method, it is greatly affected by the capacity of the sample set, and the selection of image features is relatively dependent, and the accuracy is not too high, so in combination with the subject of golf sports objects, this paper proposes a human body pose estimation algorithm based on edge contour combined with image processing. Through this algorithm, the joint points of the human body in the picture can be automatically labeled, and the posture data can be obtained. The schematic diagram is shown in Figure4.Figure 4
Schematic diagram of attitude estimation based on the image processing method.Firstly, the Canny edge detection algorithm is used to obtain its contour edge, and then rough image processing methods such as horizontal line scanning and human body length ratio constraints are used so that the coordinates of the human body joint points are obtained to achieve the purpose of posture estimation. The specific content includes the following parts. First, Gaussian filter is used to smooth the image. Image Gaussian filtering can be implemented in two ways: two one-dimensional Gaussian kernels are weighted twice, respectively, and one two-dimensional Gaussian kernel is convolved once. The equation of the one-dimensional Gaussian kernel function is as follows:(1)K=12πσex∗x/2σ∗σ.The parameters are determined to get a one-dimensional kernel vector. The equation of the two-dimensional Gaussian kernel function is as follows:(2)K=12πσ∗σex∗x+y∗y/2σ∗σ.The parameters are determined to get a two-dimensional kernel vector. Next, the two matrices of the partial derivatives in thex and y directions of the image can be calculated by the finite difference of the first-order partial derivative, and the first-order difference can be used to approximate the gradient of the image gray value. The equation of the image gray value gradient (including amplitude and direction) is shown as follows:(3)H1=−1−111,H2=1−11−1,ϕ1m,n=fm,n∗H1x,y,ϕ2m,n=fm,n∗H2x,y,ϕm,n=ϕ12m,n+ϕ22m,n,θϕ=tan−1ϕ2m,nϕ1m,n.The convolution operator used in the Canny algorithm is relatively simple. For example, in the expression,H1 and H2 and ϕ and θ represent the magnitude and direction of the edge gradient, respectively.The position of the edge cannot be completely determined based on the global gradient alone, and nonmaximum suppression of the gradient amplitude is needed to highlight the true edge. This process is an important step for edge detection. The principle of nonmaximum suppression is described as follows (as shown in Figure5).Figure 5
Schematic diagram of nonmaximum suppression.The gradient direction of pointC is shown by the blue line, and its maximum local gray scale is on this line. The intersection of the eight-valued field of point C and the gradient direction is d1 and d2. Comparing the gray value of point C in its eighth whether there is a local maximum in the value field, it can be judged whether point C is an edge point. If the gray value of point C is less than any one of d1 and d2, it means that point C is not a local maximum and point C is not an edge point. Set its gray value to 0.Finally, the double threshold method can reduce the number of false edges. Choose two thresholds: a high threshold and a low threshold. According to the two thresholds, two threshold images are obtained, respectively. The high threshold image obtains an edge image with few false edges, and then the low threshold image is used to connect the high threshold processed edge images into contours so that all edge gaps are closed.


## 3.2.1. Model-Based Human Pose Estimation

The model-based human pose estimation method is related to the constructed human model and the prior knowledge of related motions; that is, the system state at each moment is determined by the system state at the previous moment, and the projection of the human body model and the characteristics of each part of the human body in the image are measured. Between the likelihoods, find the optimal matching method to obtain the human body pose parameters. The process of model-based pose estimation methods is generally “prediction, matching, and update”; that is, first establish a human body model (skeleton model, tree diagram model, dynamic model, and so on) and then pass the previous frame of the human body based on the knowledge of the dynamic model. The posture estimation result assumes the posture of the human body in the current frame. Since the initial frame does not have the prediction of the previous frame, the posture needs to be initialized first and finally the postulated human posture result is projected into the actual image plane, and the posture evaluation function is used as the result of revising the hypothesis. The following will specifically introduce the three important components of the model-based method: building a human body model, selecting an observation matching function, and designing a prediction model.The establishment of a human body model is based on the preparation stage of the model. The human body model established, based on prior knowledge, can effectively constrain the feature change space, thereby improving the calculation efficiency and reducing the search range for matching. Human body structure models are divided into one-dimensional, two-dimensional, and three-dimensional models. In the process of human body modeling, commonly used models include skeleton models, silhouette models, three-dimensional skeleton models, and graphic structure models. The skeleton model is the simplest way to represent the structure of the human body. It is mainly composed of points and line segments, representing joint points and bones, respectively, as shown in Figure3. The skeleton model is mainly composed of points and lines, representing the joint points and bones of the human body, respectively. It is the simplest representation method of the human body structure, as shown in Figure 3(a).Figure 3
Human structure model diagram.
(a)(b)(c)In the model-based human body pose estimation method, the observation process is to project a three-dimensional human body model onto the image plane and find the optimal human body poses (parameters) by measuring the likelihood between the projected model and the real image. Commonly used image features in the observation process include optical flow, color, texture, edge, and contour.In the model-based human pose estimation method, the purpose of the prediction process is mainly to reduce the feasible and search space of the pose. Model-based human pose estimation is a process of searching and solving in a space containing a large number of poses to obtain the pose that best matches the current image pose. However, since the three-dimensional pose of the human body is mostly a high-dimensional continuous variable, what is the current challenge? Dense searching, sampling, and solving are realized in the high-dimensional pose space. Many researchers use Gaussian processes to build dynamic models based on the theory that human motion is a continuous change process with time-series relationships and finally realize predictive search in a low-dimensional space. The specific process is to use low-dimensional subspaces to represent a large number of constraints in the process of human movement and use various dimensionality reduction methods to achieve the purpose of reducing the dimensionality of the posture vector.


## 3.2.2. Human Pose Estimation Based on Contour Edge Features

The use of model-based human pose estimation methods must search in high-dimensional space. The optimization speed is slow, and it is difficult to achieve real-time processing. It has strong adaptability to the model-free human pose estimation methods based on statistical learning, and the results are accurate. However, the algorithm is more complicated, and the time cost of modeling and the labor cost of training samples are relatively high. Based on the matching model-free human body pose estimation method, it is greatly affected by the capacity of the sample set, and the selection of image features is relatively dependent, and the accuracy is not too high, so in combination with the subject of golf sports objects, this paper proposes a human body pose estimation algorithm based on edge contour combined with image processing. Through this algorithm, the joint points of the human body in the picture can be automatically labeled, and the posture data can be obtained. The schematic diagram is shown in Figure4.Figure 4
Schematic diagram of attitude estimation based on the image processing method.Firstly, the Canny edge detection algorithm is used to obtain its contour edge, and then rough image processing methods such as horizontal line scanning and human body length ratio constraints are used so that the coordinates of the human body joint points are obtained to achieve the purpose of posture estimation. The specific content includes the following parts. First, Gaussian filter is used to smooth the image. Image Gaussian filtering can be implemented in two ways: two one-dimensional Gaussian kernels are weighted twice, respectively, and one two-dimensional Gaussian kernel is convolved once. The equation of the one-dimensional Gaussian kernel function is as follows:(1)K=12πσex∗x/2σ∗σ.The parameters are determined to get a one-dimensional kernel vector. The equation of the two-dimensional Gaussian kernel function is as follows:(2)K=12πσ∗σex∗x+y∗y/2σ∗σ.The parameters are determined to get a two-dimensional kernel vector. Next, the two matrices of the partial derivatives in thex and y directions of the image can be calculated by the finite difference of the first-order partial derivative, and the first-order difference can be used to approximate the gradient of the image gray value. The equation of the image gray value gradient (including amplitude and direction) is shown as follows:(3)H1=−1−111,H2=1−11−1,ϕ1m,n=fm,n∗H1x,y,ϕ2m,n=fm,n∗H2x,y,ϕm,n=ϕ12m,n+ϕ22m,n,θϕ=tan−1ϕ2m,nϕ1m,n.The convolution operator used in the Canny algorithm is relatively simple. For example, in the expression,H1 and H2 and ϕ and θ represent the magnitude and direction of the edge gradient, respectively.The position of the edge cannot be completely determined based on the global gradient alone, and nonmaximum suppression of the gradient amplitude is needed to highlight the true edge. This process is an important step for edge detection. The principle of nonmaximum suppression is described as follows (as shown in Figure5).Figure 5
Schematic diagram of nonmaximum suppression.The gradient direction of pointC is shown by the blue line, and its maximum local gray scale is on this line. The intersection of the eight-valued field of point C and the gradient direction is d1 and d2. Comparing the gray value of point C in its eighth whether there is a local maximum in the value field, it can be judged whether point C is an edge point. If the gray value of point C is less than any one of d1 and d2, it means that point C is not a local maximum and point C is not an edge point. Set its gray value to 0.Finally, the double threshold method can reduce the number of false edges. Choose two thresholds: a high threshold and a low threshold. According to the two thresholds, two threshold images are obtained, respectively. The high threshold image obtains an edge image with few false edges, and then the low threshold image is used to connect the high threshold processed edge images into contours so that all edge gaps are closed.


## 3.3. Auxiliary Training System Based on Human Body Pose Estimation Algorithm

Traditional sports training often relies on manual experience guidance and unable to give accurate numerical analysis of key movements. This article will build an auxiliary training system based on the method of human posture estimation for golf. The system takes the swing big data of the trainer and the coach as input and uses the five joint angle trajectory comparison graphs and the posture similarity of the trainer and the coach as input. Auxiliary training indicators are combined with artificial intelligence technology [3, 4] to output posture prediction results. The frame diagram of the auxiliary sports training system built in this paper is shown in Figure 6.Figure 6
Auxiliary training system frame.Auxiliary sports training system function takes the big data image sequence of the trainer and the coach’s swing as input and takes the 5 joint angle trajectory comparison chart of the trainer and the coach and the posture similarity change chart of each frame as the auxiliary training index, compares the difference between the joint angles of the trainer and the coach, and at the same time uses artificial intelligence technology to modify the trainer’s movements according to the similarity of the posture so as to achieve the purpose of analyzing the sports posture and providing training suggestions.


## 4. Experiments

In this section, first, we discuss our experimental environment that involves real-time deployment followed by the implementation steps and the obtained experimental results.


### 4.1. Experimental Environment

The hardware and software components along with the deployed environment are shown in Table1.Table 1
Experimental hardware platforms and software simulation environment.
CPUIntel(R) Core(TM) i5-4200M CPU @ 2.50 GHzRAM16.00 GBOperating systemWindows10Development environmentVS2018Development toolsOpenCV + MATLAB


### 4.2. Implementation Steps

Step 1. The big image data or other training images collected by the Kinect sensor are used as the input of the auxiliary training systemStep 2. The swing posture data of the trainer and the coach are obtained through the target detection based on artificial intelligence and the human body posture estimation method based on the edge feature of the human body combined with image processingStep 3. 5 joint angles Angel (head, neck, and chest), Angel2 (left shoulder, left elbow, and left wrist), Angel3 (right shoulder, right hour, and right wrist), Angel4 (left marrow, left knee, and left foot), and Angle5 (right hip, right knee, and right foot) are adopted and are used as the system index, and Euclidean distance is selected as the similarity measure of attitude


### 4.3. Experimental Results

The 48 frames of swing images of the trainer and the coach are selected as the input of the system, and the output is the trajectory comparison of the joint angles and the similarity of each frame. The following is a demonstration of auxiliary training based on the joint angle trajectory. Figure7 shows a movement trajectory diagram of five joint angles. Here, the blue line represents the joint angle change diagram of the coach, and the red line represents the joint angle change diagram of the athlete. It can be seen from Figure 7 that the algorithm in this paper has achieved gratifying results. We also have compared with the gc-lstm method. Figures 8 and 9 show the effect of dropout on our model performance and reflect the comparison results of the learning curve. In Figure 8, the loss is plotted against the impact of using dropout on model performance, whereas in Figure 9, the loss is computed for different learning curves.Figure 7
Experimental comparison results.Figure 8
The impact of using dropout on model performance.Figure 9
Comparison of learning curve results.


## 4.1. Experimental Environment

The hardware and software components along with the deployed environment are shown in Table1.Table 1
Experimental hardware platforms and software simulation environment.
CPUIntel(R) Core(TM) i5-4200M CPU @ 2.50 GHzRAM16.00 GBOperating systemWindows10Development environmentVS2018Development toolsOpenCV + MATLAB


## 4.2. Implementation Steps

Step 1. The big image data or other training images collected by the Kinect sensor are used as the input of the auxiliary training systemStep 2. The swing posture data of the trainer and the coach are obtained through the target detection based on artificial intelligence and the human body posture estimation method based on the edge feature of the human body combined with image processingStep 3. 5 joint angles Angel (head, neck, and chest), Angel2 (left shoulder, left elbow, and left wrist), Angel3 (right shoulder, right hour, and right wrist), Angel4 (left marrow, left knee, and left foot), and Angle5 (right hip, right knee, and right foot) are adopted and are used as the system index, and Euclidean distance is selected as the similarity measure of attitude


## 4.3. Experimental Results

The 48 frames of swing images of the trainer and the coach are selected as the input of the system, and the output is the trajectory comparison of the joint angles and the similarity of each frame. The following is a demonstration of auxiliary training based on the joint angle trajectory. Figure7 shows a movement trajectory diagram of five joint angles. Here, the blue line represents the joint angle change diagram of the coach, and the red line represents the joint angle change diagram of the athlete. It can be seen from Figure 7 that the algorithm in this paper has achieved gratifying results. We also have compared with the gc-lstm method. Figures 8 and 9 show the effect of dropout on our model performance and reflect the comparison results of the learning curve. In Figure 8, the loss is plotted against the impact of using dropout on model performance, whereas in Figure 9, the loss is computed for different learning curves.Figure 7
Experimental comparison results.Figure 8
The impact of using dropout on model performance.Figure 9
Comparison of learning curve results.


## 5. Conclusion

Based on artificial intelligence and big data technology, in this paper, the human body posture estimation algorithm is applied to the auxiliary training system. We considered golf as the case study for this purpose. The method of human body posture estimation has completed the construction of the golf auxiliary training system, which brings standardized analysis and quantification to sports training guide. The big data image collected by the Kinect sensor is used as the input of the auxiliary training system, and the swing posture data of the trainer and the coach are obtained through methods such as target detection based on artificial intelligence and posture estimation based on the edge feature of the human body combined with image processing. Finally, the system uses two auxiliary indicators of five joint angle trajectories and posture similarity as output. By comparing the joint angle trajectories of the trainer and the coach, numerous intuitive analyses are performed on the posture similarity. In the future, we aim to apply our algorithm for various sport events to see its performance on large scale with dynamic having diversified nature.



---

*Source: 9929650-2021-05-24.xml*