# Knee Bending Motion Extraction Algorithm for Ball Motion Image Using Vision Sensor

Authors: Zhanshu An; Zhigang Yuan; Xu Cao; Shijia Song
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

To improve the extraction accuracy of knee bending motion in ball motion image and reduce the extraction distance error and time consumption, a knee bending motion extraction algorithm using visual sensor is proposed. The visual sensor model is constructed based on the ball motion frame image, the trigger data is output through differential and logical judgment, and these data are normalized to generate the visual sensor sample set of the ball motion frame image. The sample set is used as the input of the convolution neural network (CNN) and the sample basis of the motion energy model. The CNN extracts the features of the sample set in the convolution layer, the motion energy model is combined with the local binary pattern to extract the features of the sample set, the weighted summation method is used to fuse the two features, and the Softmax classifier is used to classify and extract the knee bending motion. The results show that the proposed algorithm has good ball motion image collection effect, the knee bending motion extraction accuracy is always maintained at about 98%, the distance error is low, and the time consumption of ball motion feature extraction is only 2.65 s, which has high application value in the field of sports.


---

## Body

## 1. Introduction

With the vigorous development of China’s sports industry, all kinds of ball motion have made gratifying achievements in competitive competitions. However, the body injury of athletes is also becoming more and more serious. Therefore, collecting the features of athletes’ knee bending motion by scientific means is conducive to the relevant departments to formulate the training plan of ball players [1]. In the existing research, many scholars have made relevant research. Literature [2] proposed a motion extraction algorithm based on image length and Doppler broadening. This algorithm can extract motion, but the accuracy still needs to be further improved. Literature [3] takes the time channel of video as the research object to construct an action extraction algorithm. The algorithm can consider the features of video image and realize the extraction of motion action, but the extraction effect of knee bending action is not good. Literature [4] takes the video data set as the research sample to carry out motion extraction, but similar to the above algorithm, the accuracy of extracting detailed actions is not high and still needs to be further improved. Literature [5] uses the method of deep learning to extract motion, but the algorithm does not achieve good results in the image processing stage, and the image definition is insufficient, resulting in the impact of extraction accuracy. Literature [6] uses semisupervised learning to realize action extraction. The algorithm is still in the research stage and needs further verification. The vision sensor is actually a bionic device, which is conducive to the event driven action during motion extraction, and the sensor has high dynamic range and superior comprehensive performance. Based on the visual sensor, this paper studies ball motion and constructs a special algorithm for extracting knee flexion, which points out the direction for the formulation of athletes’ training plan in the future. In order to further improve the knee bending motion extraction accuracy of ball motion image and reduce the extraction distance error and time consumption, a knee bending motion extraction algorithm based on visual sensor is designed. The main contributions of this paper are as follows: (1) Vision sensor is a bionic device, which is conducive to the event driven action during motion extraction. The sensor has high dynamic range and superior comprehensive performance, which can improve the efficiency and accuracy of ball motion image sample collection. (2) Using CNN to extract the features of ball motion images can lay a solid foundation for the subsequent knee flexion extraction. (3) Softmax classifier is used to classify and extract knee bending motion, which solves the problem of fuzzy category in the process of knee bending motion extraction of ball motion image.


## 2. Methodology

### 2.1. Construction of Sample Set Model Based on Visual Sensor

In order to accurately extract the knee bending action in the ball motion image, a visual sensor model is constructed based on the frames in the ball motion image. The imaging process of the model is shown in Figure1.Figure 1
Imaging process.According to data in Figure1, the complementary metal oxide semiconductor in the vision sensor uses the vision sensor array to sense the pixel circuit through the photodiode and quantify the light intensity [7], so that the pixels of the ball motion image are converted into electrical signals. In Figure 1, path 1 and path 2 are the imaging process of the visual sensor. The imaging principle is to add the “address event” expression as logic after each pixel unit to realize circuit judgment. Build the data set of ball motion frame image according to path 1 and path 2. In the process of building this kind of data set, it is necessary to take the pixel value and pixel information of the output ball motion frame image as the research goal and fuse an “address event” logic judgment module to the end of the visual sensor array to obtain the relevant image data set.The visual sensor itself has superior event triggering ability, but when the visual sensor triggers an event, it mainly uses the analog circuit in the visual sensor to quantify, accumulate, and release the light intensity. In this process, it can only ensure that each pixel unit in the image can work independently [8, 9]. However, there is no concept of frame and high delay. In order to solve these problems and ensure that the ordinary ball motion frame image can use the visual sensor, this study takes the ball motion frame image as the object, discretizes the pixel units in the continuous ball motion frame image through the visual sensor, and ensures that each pixel unit in the motion frame image is independent of each other. That is, the differential and logical judgment is only for the current frame and the initial state of the pixel. The logical judgment process of the visual sensor data set model of the ball motion frame image is to judge whether the light intensity change amplitude of each pixel unit is close to the trigger threshold T, and the judgment is based on the current frame and the state of the previous frame of the pixel unit.The premise of event triggering is that the change amplitude of spherical moving image pixel unit has approached the thresholdT, and Equation (1) is the expression of this case:
(1)Icx,y=Ipx,y∗1±T,where Icx,y and Ipx,y are used to express two light intensities (original and current frame), respectively. x,y is used to describe pixel unit coordinates. Ipx,y represents the light intensity of the pixel in the current frame.The visual sensor data set model method of ball motion frame image is constructed in this paper. When processing ball motion frame image, the event trigger does not occur immediately when the pixel light intensity changes close to the threshold. Generally, the event is triggered after the logical judgment of the next frame is completed. The specific process is as follows:(1)
In the process of model construction, the first frame of the ball motion image is set as the pixel light intensity in the original state, and the difference is made, and the pixel light intensity in the original state is logically judged to be the pixel light intensity in the next frame. If the result does not reach the trigger threshold, there is no need to activate the pixel unit(2)
Continue the difference and logically judge the light intensity of the original state pixel and the third frame light intensity of the same pixel(3)
According to the above rules, continue to differentiate and logically judge the light intensity of each frame. When reaching the frame, the obtained pixel unit light intensity first realizes the trigger threshold and outputs the trigger state. The pixel light intensity of the frame becomes the pixel light intensity of the new original state after output. Make a logical judgment on the pixel light intensity of subsequent frames according to the above steps, and the change amplitude of pixel light intensity isΔIx,y, as shown in Equation (2).
(2)ΔIx,y=log2Icx,yIpx,y−1≥T,Ip=Ic,Activateoutput<T,Ip=Ip,Nooutput.If the trigger thresholdT is equal to the change amplitude ΔIx,y, the pixel light intensity of the current frame of the ball moving image and the pixel light intensity of the original state can be output, and the pixel activation information can be output at the same time [10].If you want to obtain the sample set of knee bending motion visual sensor of ball motion frame image, you need to first obtain the knee bending motion trigger data from the massive frame images of ball motion continuous video stream and first disperse the continuous video stream according to the time features to obtain the ball motion frame image. After that, the ball motion frame image is classified according to the cluster, the dynamic difference and logical judgment are completed in the cluster, the dynamic active region image is obtained, and these regions are converted into time address/frame data [11, 12].Extracting knee flexion from ball motion images requires CNN for classification, but the inputs of such networks are static images [13], rather than the dynamic active area images obtained above. Therefore, this study establishes a static image simulation motion model. Random motion simulates the knee bending action in the static image and simulates the dynamic state of knee bending action by moving or enlarging and reducing the image. The static image simulation motion model is used to expand the static image sample set, and the visual sensor data set model difference and logic judgment of the ball motion frame image are used to determine the static image to simulate the motion image, and finally, the visual sensor sample set of the ball motion frame image is obtained.


### 2.2. Knee Bending Action Extraction

In order to extract the knee bending motion from the visual sensor samples of the ball motion frame image, while increasing the accuracy of extraction, the two parts of the CNN and the motion energy model algorithm are merged, and finally, the knee bending motion extraction of the ball motion image is completed.


#### 2.2.1. Feature Extraction Based on CNN

The CNN includes an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. The visual sensor sample set of the ball motion frame image obtained is the input of the input layer, and the feature extraction is mainly in the convolution. The CNN used in this paper contains 10 convolutional layers in total. Ball motion images are randomly selected from the visual sensor sample set of ball motion frame images. Each image contains multiple neurons. A feature of each image is extracted through convolution filter in each convolution layer [14–16]. The convolution process is as follows:Equation (3) describes the input of the second neuron in layer l:
(3)al=fwl∗al−1+bl,where b represents bias and w represents filter. The expression of cross entropy loss function is as follows:
(4)Jy,fx,θ=−∑i=1Cyilogfix,θ,where y is used to express the forecast target category, C is the set number of target categories, and yc=1 and fx,θ represent the probability of each category.The gradient descent method is used for iterative solution, and the gradient of the convolution layer is calculated by Equation (5):
(5)∂JW,b;x,y∂bl,k=∑i,j∂l,ki,j,where x and k represent constant and dimension, respectively, W represents weight, and i and j represent different gradient descent parameters.In this study, a total of 10 convolution layers are used, and the pool layer is between each two convolution layers. According to the principle of local correlation, the pool processing of the images in the data set obtained can reduce the computational complexity and ensure that the original ball motion image will not be deformed after feature extraction. The full connection layer including 4107 neurons is used in this paper, and several different types of features are fused. The Softmax function is used in classification. This function realizes probability mapping in multiple categories. Thex input (visual sensor sample set of ball motion frame image) is described by K-dimensional vector. The mapping form is described by Equation (6):
(6)z^=softmaxx=expx∑i=1Kexpx.


#### 2.2.2. Feature Extraction Based on Motion Energy Model

In the same ball motion, the similarity of knee bending motions of the same athlete is high. If the sample set features are extracted at one time, the extraction accuracy of knee bending motion is low [17]. In order to further increase the accuracy of knee bending motion extraction from ball motion images, we can try to segment the visual sensor sample set of ball motion frame images from dimensions to analyze the details of knee bending motion, so as to improve the effect of knee bending motion extraction [18, 19]. In this study, the visual sensor sample set of ball motion frame image is adaptively divided into several subsample sets according to the motion energy.The three-dimensional depth information is used to obtain the energy of human body in ball motion, so as to obtain the knee bending action features of three-dimensional effect. Based on the visual sensor sample set of ball motion frame image, a new three-dimensional coordinatex,y,z is constructed. z is the depth value in the right knee coordinate system. For each frame image in the visual sensor sample set based on spherical motion frame image, complete the projection to three orthogonal Cartesian planes to obtain x−y,y−z,x−z. Thus, three two-dimensional projection images of front view f, top view s, and side view t are obtained, and the frame index of the visual sensor sample set used to describe the ball motion frame image is represented by mapvi,v∈f,s,t. The motion energy model is as follows:
(7)DMM=∑i=aendmapvi−mapvi−1.When using the motion energy model to extract features, it is necessary to segment the image [20], so as to obtain the depth energy of several different frames:
(8)DMMi=nLDMM,(9)n=1,2,3,⋯,L−1,where DMMi represents the energy of the ball moving image of i frame and L stands for frame image sequence. The motion energy model cannot describe the local information of the moving image. Therefore, it is necessary to use the local binary mode LBP to extract the local features of the moving image [21, 22]. LBP operator can only cover part of the sphere moving image, but not all the frequency textures. Therefore, the extended LBP neighborhood is a circular neighborhood with R as the radius, in which there are many pixels. Suppose there is a pixel a coordinate is x,y. The neighborhood coordinates of the pixel are described as Equation (10):
(10)xi,yi=x+Rcos2πin,y−Rsin2πin.Takinga as the central pixel, set the pixel as the threshold, compare the threshold with the gray value of adjacent pixels, and obtain the n-bit binary number. The value is calculated by Equation (11).
(11)LBPn,R=∑i=0n−1Thfxi,yi−fx,y∗2i,(12)Thx=0x<01x≥0,where Th⋅ represents the image gray threshold and fxi,yi and fx,y represent the gray image and the original image, respectively.In order to obtain the texture information of ball motion image, binary mode needs to be used, and LBP feature is used to encode the motion energy model [23]. The detailed process is as follows: (1) the motion energy model template is divided into uniform grids, and the local binary mode features of each grid are counted; (2) concatenating each local binary pattern feature to obtain a local binary pattern feature descriptor; and (3) the row vector model is used to connect multiple motion energy model feature descriptors in series to obtain the final feature.


#### 2.2.3. Algorithm of Feature Fusion Knee Bending Motion Extraction

In this study, the algorithm combines CNN and motion energy model and finally realizes the knee bending motion extraction of ball motion. Figure2 shows the process of the proposed algorithm.Figure 2
The proposed algorithm process.According to Figure2, there are four key steps to realize knee bending action extraction:
(1)
CNN Feature Extraction. Take the visual sensor sample set of ball motion frame image as the input, set the size of each image in the sample set as 224 ∗ 224, feature extraction is carried out in the convolution layer, set 3 ∗ 3 as the convolution kernel size, and use back propagation and cross entropy loss function when optimizing CNN parameters(2)
Feature Extraction of Motion Energy. The energy of human for ball motion is obtained by using three-dimensional depth information, the three-dimensional coordinate system is constructed based on the visual sensor sample set of ball motion frame image, the motion energy model is constructed, and the features are obtained by using local binary mode(3)
Fuse the Features of the Two Algorithms. The weighted summation method is used in feature fusion. Set Z=X1,X2,⋯,Xm to describe the weight vector, and Equation (8) describes the feature fusion Yfusion model:(4)
The visual sensor sample set of ball motion frame image is divided into training set, verification set, and test set. The training set is used to train the feature fusion model, and the training of the completed model is obtained(5)
In the model, the verification set and test set are input, respectively, to obtain the model parameters and determine the model structure. The knee bending motion extraction results of ball motion images are obtained by using the verification set


## 2.1. Construction of Sample Set Model Based on Visual Sensor

In order to accurately extract the knee bending action in the ball motion image, a visual sensor model is constructed based on the frames in the ball motion image. The imaging process of the model is shown in Figure1.Figure 1
Imaging process.According to data in Figure1, the complementary metal oxide semiconductor in the vision sensor uses the vision sensor array to sense the pixel circuit through the photodiode and quantify the light intensity [7], so that the pixels of the ball motion image are converted into electrical signals. In Figure 1, path 1 and path 2 are the imaging process of the visual sensor. The imaging principle is to add the “address event” expression as logic after each pixel unit to realize circuit judgment. Build the data set of ball motion frame image according to path 1 and path 2. In the process of building this kind of data set, it is necessary to take the pixel value and pixel information of the output ball motion frame image as the research goal and fuse an “address event” logic judgment module to the end of the visual sensor array to obtain the relevant image data set.The visual sensor itself has superior event triggering ability, but when the visual sensor triggers an event, it mainly uses the analog circuit in the visual sensor to quantify, accumulate, and release the light intensity. In this process, it can only ensure that each pixel unit in the image can work independently [8, 9]. However, there is no concept of frame and high delay. In order to solve these problems and ensure that the ordinary ball motion frame image can use the visual sensor, this study takes the ball motion frame image as the object, discretizes the pixel units in the continuous ball motion frame image through the visual sensor, and ensures that each pixel unit in the motion frame image is independent of each other. That is, the differential and logical judgment is only for the current frame and the initial state of the pixel. The logical judgment process of the visual sensor data set model of the ball motion frame image is to judge whether the light intensity change amplitude of each pixel unit is close to the trigger threshold T, and the judgment is based on the current frame and the state of the previous frame of the pixel unit.The premise of event triggering is that the change amplitude of spherical moving image pixel unit has approached the thresholdT, and Equation (1) is the expression of this case:
(1)Icx,y=Ipx,y∗1±T,where Icx,y and Ipx,y are used to express two light intensities (original and current frame), respectively. x,y is used to describe pixel unit coordinates. Ipx,y represents the light intensity of the pixel in the current frame.The visual sensor data set model method of ball motion frame image is constructed in this paper. When processing ball motion frame image, the event trigger does not occur immediately when the pixel light intensity changes close to the threshold. Generally, the event is triggered after the logical judgment of the next frame is completed. The specific process is as follows:(1)
In the process of model construction, the first frame of the ball motion image is set as the pixel light intensity in the original state, and the difference is made, and the pixel light intensity in the original state is logically judged to be the pixel light intensity in the next frame. If the result does not reach the trigger threshold, there is no need to activate the pixel unit(2)
Continue the difference and logically judge the light intensity of the original state pixel and the third frame light intensity of the same pixel(3)
According to the above rules, continue to differentiate and logically judge the light intensity of each frame. When reaching the frame, the obtained pixel unit light intensity first realizes the trigger threshold and outputs the trigger state. The pixel light intensity of the frame becomes the pixel light intensity of the new original state after output. Make a logical judgment on the pixel light intensity of subsequent frames according to the above steps, and the change amplitude of pixel light intensity isΔIx,y, as shown in Equation (2).
(2)ΔIx,y=log2Icx,yIpx,y−1≥T,Ip=Ic,Activateoutput<T,Ip=Ip,Nooutput.If the trigger thresholdT is equal to the change amplitude ΔIx,y, the pixel light intensity of the current frame of the ball moving image and the pixel light intensity of the original state can be output, and the pixel activation information can be output at the same time [10].If you want to obtain the sample set of knee bending motion visual sensor of ball motion frame image, you need to first obtain the knee bending motion trigger data from the massive frame images of ball motion continuous video stream and first disperse the continuous video stream according to the time features to obtain the ball motion frame image. After that, the ball motion frame image is classified according to the cluster, the dynamic difference and logical judgment are completed in the cluster, the dynamic active region image is obtained, and these regions are converted into time address/frame data [11, 12].Extracting knee flexion from ball motion images requires CNN for classification, but the inputs of such networks are static images [13], rather than the dynamic active area images obtained above. Therefore, this study establishes a static image simulation motion model. Random motion simulates the knee bending action in the static image and simulates the dynamic state of knee bending action by moving or enlarging and reducing the image. The static image simulation motion model is used to expand the static image sample set, and the visual sensor data set model difference and logic judgment of the ball motion frame image are used to determine the static image to simulate the motion image, and finally, the visual sensor sample set of the ball motion frame image is obtained.


## 2.2. Knee Bending Action Extraction

In order to extract the knee bending motion from the visual sensor samples of the ball motion frame image, while increasing the accuracy of extraction, the two parts of the CNN and the motion energy model algorithm are merged, and finally, the knee bending motion extraction of the ball motion image is completed.


### 2.2.1. Feature Extraction Based on CNN

The CNN includes an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. The visual sensor sample set of the ball motion frame image obtained is the input of the input layer, and the feature extraction is mainly in the convolution. The CNN used in this paper contains 10 convolutional layers in total. Ball motion images are randomly selected from the visual sensor sample set of ball motion frame images. Each image contains multiple neurons. A feature of each image is extracted through convolution filter in each convolution layer [14–16]. The convolution process is as follows:Equation (3) describes the input of the second neuron in layer l:
(3)al=fwl∗al−1+bl,where b represents bias and w represents filter. The expression of cross entropy loss function is as follows:
(4)Jy,fx,θ=−∑i=1Cyilogfix,θ,where y is used to express the forecast target category, C is the set number of target categories, and yc=1 and fx,θ represent the probability of each category.The gradient descent method is used for iterative solution, and the gradient of the convolution layer is calculated by Equation (5):
(5)∂JW,b;x,y∂bl,k=∑i,j∂l,ki,j,where x and k represent constant and dimension, respectively, W represents weight, and i and j represent different gradient descent parameters.In this study, a total of 10 convolution layers are used, and the pool layer is between each two convolution layers. According to the principle of local correlation, the pool processing of the images in the data set obtained can reduce the computational complexity and ensure that the original ball motion image will not be deformed after feature extraction. The full connection layer including 4107 neurons is used in this paper, and several different types of features are fused. The Softmax function is used in classification. This function realizes probability mapping in multiple categories. Thex input (visual sensor sample set of ball motion frame image) is described by K-dimensional vector. The mapping form is described by Equation (6):
(6)z^=softmaxx=expx∑i=1Kexpx.


### 2.2.2. Feature Extraction Based on Motion Energy Model

In the same ball motion, the similarity of knee bending motions of the same athlete is high. If the sample set features are extracted at one time, the extraction accuracy of knee bending motion is low [17]. In order to further increase the accuracy of knee bending motion extraction from ball motion images, we can try to segment the visual sensor sample set of ball motion frame images from dimensions to analyze the details of knee bending motion, so as to improve the effect of knee bending motion extraction [18, 19]. In this study, the visual sensor sample set of ball motion frame image is adaptively divided into several subsample sets according to the motion energy.The three-dimensional depth information is used to obtain the energy of human body in ball motion, so as to obtain the knee bending action features of three-dimensional effect. Based on the visual sensor sample set of ball motion frame image, a new three-dimensional coordinatex,y,z is constructed. z is the depth value in the right knee coordinate system. For each frame image in the visual sensor sample set based on spherical motion frame image, complete the projection to three orthogonal Cartesian planes to obtain x−y,y−z,x−z. Thus, three two-dimensional projection images of front view f, top view s, and side view t are obtained, and the frame index of the visual sensor sample set used to describe the ball motion frame image is represented by mapvi,v∈f,s,t. The motion energy model is as follows:
(7)DMM=∑i=aendmapvi−mapvi−1.When using the motion energy model to extract features, it is necessary to segment the image [20], so as to obtain the depth energy of several different frames:
(8)DMMi=nLDMM,(9)n=1,2,3,⋯,L−1,where DMMi represents the energy of the ball moving image of i frame and L stands for frame image sequence. The motion energy model cannot describe the local information of the moving image. Therefore, it is necessary to use the local binary mode LBP to extract the local features of the moving image [21, 22]. LBP operator can only cover part of the sphere moving image, but not all the frequency textures. Therefore, the extended LBP neighborhood is a circular neighborhood with R as the radius, in which there are many pixels. Suppose there is a pixel a coordinate is x,y. The neighborhood coordinates of the pixel are described as Equation (10):
(10)xi,yi=x+Rcos2πin,y−Rsin2πin.Takinga as the central pixel, set the pixel as the threshold, compare the threshold with the gray value of adjacent pixels, and obtain the n-bit binary number. The value is calculated by Equation (11).
(11)LBPn,R=∑i=0n−1Thfxi,yi−fx,y∗2i,(12)Thx=0x<01x≥0,where Th⋅ represents the image gray threshold and fxi,yi and fx,y represent the gray image and the original image, respectively.In order to obtain the texture information of ball motion image, binary mode needs to be used, and LBP feature is used to encode the motion energy model [23]. The detailed process is as follows: (1) the motion energy model template is divided into uniform grids, and the local binary mode features of each grid are counted; (2) concatenating each local binary pattern feature to obtain a local binary pattern feature descriptor; and (3) the row vector model is used to connect multiple motion energy model feature descriptors in series to obtain the final feature.


### 2.2.3. Algorithm of Feature Fusion Knee Bending Motion Extraction

In this study, the algorithm combines CNN and motion energy model and finally realizes the knee bending motion extraction of ball motion. Figure2 shows the process of the proposed algorithm.Figure 2
The proposed algorithm process.According to Figure2, there are four key steps to realize knee bending action extraction:
(1)
CNN Feature Extraction. Take the visual sensor sample set of ball motion frame image as the input, set the size of each image in the sample set as 224 ∗ 224, feature extraction is carried out in the convolution layer, set 3 ∗ 3 as the convolution kernel size, and use back propagation and cross entropy loss function when optimizing CNN parameters(2)
Feature Extraction of Motion Energy. The energy of human for ball motion is obtained by using three-dimensional depth information, the three-dimensional coordinate system is constructed based on the visual sensor sample set of ball motion frame image, the motion energy model is constructed, and the features are obtained by using local binary mode(3)
Fuse the Features of the Two Algorithms. The weighted summation method is used in feature fusion. Set Z=X1,X2,⋯,Xm to describe the weight vector, and Equation (8) describes the feature fusion Yfusion model:(4)
The visual sensor sample set of ball motion frame image is divided into training set, verification set, and test set. The training set is used to train the feature fusion model, and the training of the completed model is obtained(5)
In the model, the verification set and test set are input, respectively, to obtain the model parameters and determine the model structure. The knee bending motion extraction results of ball motion images are obtained by using the verification set


## 2.2.1. Feature Extraction Based on CNN

The CNN includes an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. The visual sensor sample set of the ball motion frame image obtained is the input of the input layer, and the feature extraction is mainly in the convolution. The CNN used in this paper contains 10 convolutional layers in total. Ball motion images are randomly selected from the visual sensor sample set of ball motion frame images. Each image contains multiple neurons. A feature of each image is extracted through convolution filter in each convolution layer [14–16]. The convolution process is as follows:Equation (3) describes the input of the second neuron in layer l:
(3)al=fwl∗al−1+bl,where b represents bias and w represents filter. The expression of cross entropy loss function is as follows:
(4)Jy,fx,θ=−∑i=1Cyilogfix,θ,where y is used to express the forecast target category, C is the set number of target categories, and yc=1 and fx,θ represent the probability of each category.The gradient descent method is used for iterative solution, and the gradient of the convolution layer is calculated by Equation (5):
(5)∂JW,b;x,y∂bl,k=∑i,j∂l,ki,j,where x and k represent constant and dimension, respectively, W represents weight, and i and j represent different gradient descent parameters.In this study, a total of 10 convolution layers are used, and the pool layer is between each two convolution layers. According to the principle of local correlation, the pool processing of the images in the data set obtained can reduce the computational complexity and ensure that the original ball motion image will not be deformed after feature extraction. The full connection layer including 4107 neurons is used in this paper, and several different types of features are fused. The Softmax function is used in classification. This function realizes probability mapping in multiple categories. Thex input (visual sensor sample set of ball motion frame image) is described by K-dimensional vector. The mapping form is described by Equation (6):
(6)z^=softmaxx=expx∑i=1Kexpx.


## 2.2.2. Feature Extraction Based on Motion Energy Model

In the same ball motion, the similarity of knee bending motions of the same athlete is high. If the sample set features are extracted at one time, the extraction accuracy of knee bending motion is low [17]. In order to further increase the accuracy of knee bending motion extraction from ball motion images, we can try to segment the visual sensor sample set of ball motion frame images from dimensions to analyze the details of knee bending motion, so as to improve the effect of knee bending motion extraction [18, 19]. In this study, the visual sensor sample set of ball motion frame image is adaptively divided into several subsample sets according to the motion energy.The three-dimensional depth information is used to obtain the energy of human body in ball motion, so as to obtain the knee bending action features of three-dimensional effect. Based on the visual sensor sample set of ball motion frame image, a new three-dimensional coordinatex,y,z is constructed. z is the depth value in the right knee coordinate system. For each frame image in the visual sensor sample set based on spherical motion frame image, complete the projection to three orthogonal Cartesian planes to obtain x−y,y−z,x−z. Thus, three two-dimensional projection images of front view f, top view s, and side view t are obtained, and the frame index of the visual sensor sample set used to describe the ball motion frame image is represented by mapvi,v∈f,s,t. The motion energy model is as follows:
(7)DMM=∑i=aendmapvi−mapvi−1.When using the motion energy model to extract features, it is necessary to segment the image [20], so as to obtain the depth energy of several different frames:
(8)DMMi=nLDMM,(9)n=1,2,3,⋯,L−1,where DMMi represents the energy of the ball moving image of i frame and L stands for frame image sequence. The motion energy model cannot describe the local information of the moving image. Therefore, it is necessary to use the local binary mode LBP to extract the local features of the moving image [21, 22]. LBP operator can only cover part of the sphere moving image, but not all the frequency textures. Therefore, the extended LBP neighborhood is a circular neighborhood with R as the radius, in which there are many pixels. Suppose there is a pixel a coordinate is x,y. The neighborhood coordinates of the pixel are described as Equation (10):
(10)xi,yi=x+Rcos2πin,y−Rsin2πin.Takinga as the central pixel, set the pixel as the threshold, compare the threshold with the gray value of adjacent pixels, and obtain the n-bit binary number. The value is calculated by Equation (11).
(11)LBPn,R=∑i=0n−1Thfxi,yi−fx,y∗2i,(12)Thx=0x<01x≥0,where Th⋅ represents the image gray threshold and fxi,yi and fx,y represent the gray image and the original image, respectively.In order to obtain the texture information of ball motion image, binary mode needs to be used, and LBP feature is used to encode the motion energy model [23]. The detailed process is as follows: (1) the motion energy model template is divided into uniform grids, and the local binary mode features of each grid are counted; (2) concatenating each local binary pattern feature to obtain a local binary pattern feature descriptor; and (3) the row vector model is used to connect multiple motion energy model feature descriptors in series to obtain the final feature.


## 2.2.3. Algorithm of Feature Fusion Knee Bending Motion Extraction

In this study, the algorithm combines CNN and motion energy model and finally realizes the knee bending motion extraction of ball motion. Figure2 shows the process of the proposed algorithm.Figure 2
The proposed algorithm process.According to Figure2, there are four key steps to realize knee bending action extraction:
(1)
CNN Feature Extraction. Take the visual sensor sample set of ball motion frame image as the input, set the size of each image in the sample set as 224 ∗ 224, feature extraction is carried out in the convolution layer, set 3 ∗ 3 as the convolution kernel size, and use back propagation and cross entropy loss function when optimizing CNN parameters(2)
Feature Extraction of Motion Energy. The energy of human for ball motion is obtained by using three-dimensional depth information, the three-dimensional coordinate system is constructed based on the visual sensor sample set of ball motion frame image, the motion energy model is constructed, and the features are obtained by using local binary mode(3)
Fuse the Features of the Two Algorithms. The weighted summation method is used in feature fusion. Set Z=X1,X2,⋯,Xm to describe the weight vector, and Equation (8) describes the feature fusion Yfusion model:(4)
The visual sensor sample set of ball motion frame image is divided into training set, verification set, and test set. The training set is used to train the feature fusion model, and the training of the completed model is obtained(5)
In the model, the verification set and test set are input, respectively, to obtain the model parameters and determine the model structure. The knee bending motion extraction results of ball motion images are obtained by using the verification set


## 3. Experimental Analysis and Results

### 3.1. Data Set

Two data sets are selected in the experimental analysis process of this paper. Data set 1 is a ball motion data set under complex background, from UCF Sports Action. The database contains 150 video sequences with a resolution of 720∗ 480, with a frame rate of 10 fps and an average of 6.39 s for each video. The database contains a large number of images of behavior video, scene, and perspective changes. The data set 2 is a ball motion data set under a single background, which is mainly obtained in the network and captured by web crawlers. The data volume is 324 g. This paper uses the data in these two data sets to carry out the knee bending action extraction experiment of ball motion images. The dataset for evaluation is divided in such a way that 80% of the data is used for training, and 20% of the data is used for testing.


### 3.2. Experimental Index

The extraction algorithm based on image length and Doppler broadening (literature [2] algorithm), the extraction algorithm based on combining practical channel information (literature [3] algorithm), the extraction algorithm based on the video data set (literature [4] algorithm), the extraction algorithm based on deep network (literature [5] algorithm), the extraction algorithm based on semisupervised learning (literature [6] algorithm), and the algorithm in this paper are used as experimental algorithms to verify the actual application effects of different algorithms by verifying different index.Motion image collection effect: for two data sets, use the algorithm in this paper to construct a visual sensor sample set of ball motion frame images. The sample set collection needs to be obtained through the visual sensor data set model of the ball motion frame image. In order to verify the anti-interference of the image sample, it is necessary to change the image light intensity through gamma change to verify the image acquisition effect of the algorithm in this paper:
(13)Icx,y=Icc,y255λ∗255,where Icc,y represents the collected original image and λ represents the image scale.Knee bending action extraction effect: for ball motion images, the higher the image quality, the lower the false extraction of knee bending motion features, and the better the extraction effect.Knee flexion extraction accuracy: accuracy is an important index to evaluate the effect of knee flexion extracted by the algorithm. The images in data set 1 belong to complex background images, and the effect of the algorithm can be better analyzed by extracting such complex background images. Therefore, the experiment is mainly aimed at dataset 1. In the analysis process, K is used to represent the number of scenes in data set 1. The calculation process of extraction accuracy P is shown in Equation (14):
(14)P=∑i=1kCiK∗100%,where Ci represents the knee bending action to be extracted in the ith image.Distance error: distance error is an important index to measure whether there is an error between the extraction result of knee motion in ball motion image and the actual position of knee motion change in the image. The calculation equation of this index is as follows:
(15)G=kx,y−zx,y,where kx,y and zx,y represent the standard coordinates of the same knee bending position and the coordinates calculated by different algorithms, respectively.Extraction time consumption: it covers a wide range of motions. It is verified that each algorithm extracts the changes of time consumption of knee flexion before and after application for different ball motions. The calculation formula of extracted time consumption is as follows:
(16)T=te−ts,where te and ts represent the end and start time of knee flexion extraction, respectively.


## 3.1. Data Set

Two data sets are selected in the experimental analysis process of this paper. Data set 1 is a ball motion data set under complex background, from UCF Sports Action. The database contains 150 video sequences with a resolution of 720∗ 480, with a frame rate of 10 fps and an average of 6.39 s for each video. The database contains a large number of images of behavior video, scene, and perspective changes. The data set 2 is a ball motion data set under a single background, which is mainly obtained in the network and captured by web crawlers. The data volume is 324 g. This paper uses the data in these two data sets to carry out the knee bending action extraction experiment of ball motion images. The dataset for evaluation is divided in such a way that 80% of the data is used for training, and 20% of the data is used for testing.


## 3.2. Experimental Index

The extraction algorithm based on image length and Doppler broadening (literature [2] algorithm), the extraction algorithm based on combining practical channel information (literature [3] algorithm), the extraction algorithm based on the video data set (literature [4] algorithm), the extraction algorithm based on deep network (literature [5] algorithm), the extraction algorithm based on semisupervised learning (literature [6] algorithm), and the algorithm in this paper are used as experimental algorithms to verify the actual application effects of different algorithms by verifying different index.Motion image collection effect: for two data sets, use the algorithm in this paper to construct a visual sensor sample set of ball motion frame images. The sample set collection needs to be obtained through the visual sensor data set model of the ball motion frame image. In order to verify the anti-interference of the image sample, it is necessary to change the image light intensity through gamma change to verify the image acquisition effect of the algorithm in this paper:
(13)Icx,y=Icc,y255λ∗255,where Icc,y represents the collected original image and λ represents the image scale.Knee bending action extraction effect: for ball motion images, the higher the image quality, the lower the false extraction of knee bending motion features, and the better the extraction effect.Knee flexion extraction accuracy: accuracy is an important index to evaluate the effect of knee flexion extracted by the algorithm. The images in data set 1 belong to complex background images, and the effect of the algorithm can be better analyzed by extracting such complex background images. Therefore, the experiment is mainly aimed at dataset 1. In the analysis process, K is used to represent the number of scenes in data set 1. The calculation process of extraction accuracy P is shown in Equation (14):
(14)P=∑i=1kCiK∗100%,where Ci represents the knee bending action to be extracted in the ith image.Distance error: distance error is an important index to measure whether there is an error between the extraction result of knee motion in ball motion image and the actual position of knee motion change in the image. The calculation equation of this index is as follows:
(15)G=kx,y−zx,y,where kx,y and zx,y represent the standard coordinates of the same knee bending position and the coordinates calculated by different algorithms, respectively.Extraction time consumption: it covers a wide range of motions. It is verified that each algorithm extracts the changes of time consumption of knee flexion before and after application for different ball motions. The calculation formula of extracted time consumption is as follows:
(16)T=te−ts,where te and ts represent the end and start time of knee flexion extraction, respectively.


## 4. Results and Discussion

It can be seen from Figure3 that the images in the sample set constructed using the method in this paper have high definition. Ball motion image sample set is constructed in complex background or single background. The design model has strong anti-interference ability for complex background changes and high light intensity and can retain the basic information and original details in the image. The image effect after using the model in this paper is shown in Figure 3.Figure 3
Image effect.
(a)
Data set 1 original(b)
Data set 2 original(c)
Data set 1 after using the model in this paper(d)
Data set 2 after using the model in this paperAt the same time, the knee bending action of ball motion image in the data set is extracted. The comparison results of each comparison algorithm and the algorithm in this paper are shown in Figure4.Figure 4
Comparison of extraction effects of different algorithms.
(a)
Literature [2] algorithm extracts the results of data set 1 and data set 2(b)
Literature [3] algorithm extracts the results of data set 1 and data set 2(c)
Literature [3] algorithm extracts the results of data set 1 and data set 2(d)
Literature [4] algorithm extracts the results of data set 1 and data set 2(e)
Literature [5] algorithm extracts the results of data set 1 and data set 2(f)
Literature [6] algorithm extracts the results of data set 1 and data set 2It can be seen from Figure4 that there are great differences in the knee bending action extraction effect in the ball motion image of each comparison algorithm. The image processing in Figures 4(c) and 4(d) has poor quality and insufficient clarity and brightness. Therefore, there is also a large error in the knee bending action extraction result. Instead of accurately extracting the knee bending action in ball motion, the nonknee area is extracted; the probability of missing extraction and false extraction is high. The cleaning angle after image processing of the algorithms in Figures 4(a), 4(b), and 4(e) is moderate, but the effect of extracting the knee bending action is still not very ideal. For example, the extraction algorithm in literature [5] in Figure 4(a) cannot distinguish between the foreground and the background, and some pixels in the complex background are also extracted. The false extraction is serious, but for the image with a single background, there are extraction errors; that is, the false recognition is serious, so it still needs to be further improved. This algorithm has been able to obtain high-quality images in the sample construction stage. Therefore, the knee bending motion extraction effect is also more accurate; there is no false extraction or missing extraction and has a better extraction effect. Calculate the different situations of the number of images in the data set. Each algorithm extracts the accuracy rate of the knee bending action, and the calculation result statistics are shown in Figure 5.Figure 5
Accuracy calculation results.It can be seen from Figure5 that although the five comparison algorithms can achieve the extraction of knee bending motions, as the amount of data increases, the extraction accuracy gradually decreases. The extraction accuracy of the algorithm in this paper is always 98%. Therefore, the extraction accuracy of the knee bending motion of the ball motion image of the algorithm in this paper is higher, and the effect is better.After each algorithm processes the ball motion image, the distance error statistics are shown in Figure6.Figure 6
Comparison results of distance error.It can be seen from Figure6 that the distance error of the algorithm in literature [2] is about 20 pixels, the distance error of the algorithm in literature [3] is about 33 pixels, and the distance error of the algorithm in literature [4] is about 55 pixels, which is the highest among the six algorithms; the distance error of the algorithm in literature [5] is about 40 pixels. The distance error of the algorithm in literature [6] is about 23 pixels, while the distance error of the algorithm in this paper is about 10 pixels. The distance error of the algorithm in this paper is the smallest, which further verifies that the algorithm in this paper can more accurately extract knee bending motions in ball motion images.Ball motions cover a wide range, verify that each algorithm is used for different ball motions, before and after application, and the time consumption of the knee bending action is extracted. The comparison test results are shown in Table1.Table 1
Extraction time consumption before and after algorithm application.
Ball sports categoryAlgorithmsTime consumption before application/STime consumption after application/SVolleyballLiterature [2]11.596.14Literature [3]6.66Literature [4]7.42Literature [5]7.51Literature [6]5.83This paper2.65BasketballLiterature [2]15.799.84Literature [3]8.43Literature [4]10.47Literature [5]11.63Literature [6]7.21This paper2.89FootballLiterature [2]13.618.61Literature [3]7.69Literature [4]11.84Literature [5]10.66Literature [6]6.95This paper1.39It can be seen from Table1 that for different ball motions, compared with the literature algorithm, the algorithm in this paper can effectively shorten the time required to extract the knee bending action, and the calculation process has high efficiency. For example, for volleyball, the time before the algorithm is applied is 11.59 s. In literature [2], the time consumption after the algorithm is applied is 6.14 s; in literature [3], the time consumption after the algorithm is applied is 6.66 s; in literature [4], the time consumption after the algorithm is applied is 7.42 s; in literature [5], the time consumption after the algorithm is applied is 7.51 s; and in literature [6], the time consumption after the algorithm is applied is 5.83 s. Compared with these algorithms, the time consumption of the algorithm in this paper is only 2.65 s after application, which shows that the algorithm in this paper can effectively shorten the time required to extract the knee bending action, and the overall efficiency of the algorithm is high.


## 5. Conclusions and Future

This paper studies the knee bending motion extraction algorithm of ball motion images based on visual sensors and builds a sample set of ball sports images on the basis of visual sensors. Based on the sample set, the motion energy model and CNN are used to extract the features. The feature results are fused in the CNN, and finally, the knee bending motion is extracted. Experiments show that the algorithm has high accuracy and short extraction time. Compared with similar algorithms, it has better performance and better practical application effect. However, the algorithm in this paper still has some defects; that is, the amount of experimental data is less. Therefore, the future work needs to take this problem as the goal to test the application performance of the algorithm.



---

*Source: 4411456-2022-03-08.xml*