# Data-Driven Computer Choreography Based on Kinect and 3D Technology

Authors: Muyuan Ma; Shan Sun; Yang Gao
Journal: Scientific Programming
 (2022)

---

## Abstract

As a form of artistic expression, dance accompanied by music enriches the cultural life of human beings and stimulates the creative enthusiasm of the public. Choreography is usually done by professional choreographers. It is highly professional and time-consuming. The development of technology is changing the way of artistic creation. The development of motion capture technology and artificial intelligence makes computer-based automatic choreography possible. This paper proposes a method of music choreography based on deep learning. First, we use Kinect to extract and filter actions and get actions with high authenticity and continuity. Then, based on the constantQ transformation, the overall note density and beats per minute (BPM) of the target music are extracted, and preliminary matching is performed with features such as action speed and spatiality, and then, the local features of the music and action segments based on rhythm and intensity are matched. The experimental results show that the method proposed in this paper can effectively synthesize dance movements. The speed and other characteristics of each movement segment in the synthesis result are very uniform, and the overall choreography is more aesthetic.


---

## Body

## 1. Introduction

Movement is the soul of dance. The first problem that needs to be solved in computer choreography is how to digitize dance movements. Kinect is a somatosensory device released by Microsoft in 2010. Its working principle is to automatically recognize the joint motion data of the human skeleton through infrared rays and capture the skeleton of human motion. Kinect is known as the epoch-making work of the third generation of human-computer interaction technology. In recent years, the use of Kinect for action extraction and then realization of computer dance art has become a research hotspot. In the display of dance, the traditional method is that the choreographer verbally explains or tells the performer by drawing a picture, and then, the dancer shows the specific movement, and subsequent modifications need to be carried out on the actual effect. This requires many repetitive performances by the actors, and the workload is heavy. The use of computer for choreography can show the choreography dance through 3D virtual reality technology so that the choreographer can create, arrange, modify, and preview the effect in the computer in advance. Based on the above technology, it is possible to make simple choreography using a computer.Many existing motion generation technologies based on machine learning have been applied to dance research, including dimension reduction technology [1], Gaussian process [2], and hidden Markov model [3], so as to capture the potential correlation between music and dance motion characteristics. Dimension reduction technology can map the high-dimensional features of motion to low-dimensional space, so as to capture the potential correlation behind joint rotation in motion capture data [4]. However, the algorithm needs preprocessing steps, such as sequence alignment and fixed data length, and cannot directly model the timeliness of motion data, which limits its application in real dance motion data. Gaussian process late variable models can effectively summarize the changes of human cloud force, but they are not suitable for real-time generation because they require a lot of computing and memory resources [5]. HMM model overcomes the limitations of the two types of models mentioned above, but its ability to capture data changes is limited [6]. In order to solve the defects of computer choreography based on machine learning, this paper will introduce deep learning method to improve the novelty and coherence of generated actions.


## 2. Methods

### 2.1. Overview

First of all, the problem that needs to be solved is motion capture based on Kinect. When Kinect captures human body motion data, it captures 20 joint nodes of the human body through infrared rays. As long as you face Kinect and do dance movements, you can record the corresponding raw data. This avoids the need to place sensors on the capturer during traditional motion capture, which hinders the fluency of dance movements. Convert the captured raw data into the BVH file format to save the dance moves. The motion data file in BVH format is one of the standard file formats in the motion capture industry and is widely supported by mainstream animation production software. When using Kinect to collect dance movements, it is conducive to the optimal display and file storage of virtual dances based on the “Principle of Optimal Movements” and “Principle of Minimal Data Simplification” [7].Secondly, it is necessary to edit and integrate dance movements. Using the 3D software MAYA as the platform, the basic dance moves stored in the BVH format are imported into MAYA for editing. The choreographer can make modifications and combinations based on the existing basic dance moves, such as modifying the angle of the limbs, the speed of a certain movement, and the number of repetitions [8]; at the same time, they can also arrange and integrate the sequence of specific dance moves to express different dance themes. In addition, the choreography of dance emphasizes innovation. Using Kinect equipment, you can capture newly designed dance moves at any time. After converting it into BVH format, it becomes the new basic action element.Finally, it is necessary to match the skeleton and the character and realize the three-dimensional display. After the preliminary editing of a certain dance movement is completed, the 3D characters can be driven to dance. MAYA provides the function of binding the skeleton to the model so that the 3D model follows the movement of the skeleton. Modify the bound model by adjusting the weight of the model, the size of the skeleton, and other factors so that it can express the dance effect as realistically as possible. For the choreography of large-scale dances, after the choreography of a certain model is completed in advance, the skinned models can be copied and arranged directly to create the ideal formation and stage effect.


### 2.2. Motion Capture

#### 2.2.1. Two-Dimensional Key Point Recognition Network

The key points of the two-dimensional human body are identified using the Stacked Hourglass Network architecture. This network is constructed by loop nesting of the Hourglass subnetwork. The Hourglass subnetwork is composed of a basic structure called residual module. The residual module structure is shown in Figure1(a). The input value will pass through the upper and lower paths. This design is inspired by the residual network. The above path is a convolution path, which contains three convolution layers with different sizes of convolution kernels, which are represented by white rectangles in the figure. The three lines of text in the rectangle represent the number of input channels (NIn), the size of the convolution kernel (K), and the number of output channels (NOut) from top to bottom. Among the three convolutional layers, there are also batch regularization layers and nonlinear activation layers (RELUs), all of which are represented by gray rectangles. The lower path is a jumper path. This path contains only a convolutional layer with a convolution kernel size of 1 × 1, which will change the number of channels of the input value, and its output value will be directly added to the output value of the upper path.Figure 1
Two-dimensional key point recognition network. (a) Residual module structure. (b)Hourglass subnetwork structure.
(a)(b)The structure of the Hourglass subnetwork is shown in Figure1(b). Each white rectangle in the figure represents one of the above residual modules. In the upper half, feature extraction is performed continuously, and in the lower half, downsampling is performed through the maximum pooling operation first, and after several residual modules, the nearest neighbor interpolation method is used for upsampling. The two gray rectangles in the figure represent the downsampling and upsampling processes, respectively. It can be seen that there is a dashed line frame in the Hourglass subnet structure diagram. The dashed line frame is replaced by an Hourglass subnet, which is called a second-order Hourglass network. In this study, the Hourglass subnetwork is looped and nested four times for two-dimensional key point detection, namely, the fourth-order Hourglass network.


#### 2.2.2. Three-Dimensional Keypoint Regression Neural Network

After the two-dimensional human body key points are recognized, a set of two-dimensional coordinate values of the human body key points will be output. Then, use this set of two-dimensional coordinatesx∈R2n and use the neural network to return to the corresponding set of three-dimensional coordinate values y∈R3n. The mapping relationship of the three-dimensional coordinates regressed from the two-dimensional coordinates is f:R2n⟶R3n. Optimize the neural network by minimizing the following prediction errors, namely,(1)f∗=min1N∑i=1NLfxi−yi.Among them,L represents the Euclidean distance between vectors as the loss function, N represents the total number of key points to be identified, and f∗ represents a regression neural network. The structure of the regression neural network model is shown in Figure 2. First, the dimension is converted to 1024 dimensions through a fully connected layer. After that, it passes through the batch regularization layer, the RELUs layer, and the Dropout layer in turn and contains a jump-through path. The structure in the dashed frame in Figure 2 will be repeated once, and finally, a 3n-dimensional output vector will be formed through a fully connected layer. So far, the three-dimensional coordinates of the key points of the presenter are obtained.Figure 2
Three-dimensional key point regression neural network.


### 2.3. Action Recurrence

After obtaining the coordinates of the key points of the presenter, they need to be mapped to the robot. The control of the robot takes the angle of each joint as the control command, and the process of using the coordinate value of each key point to calculate the joint angle is called inverse kinematics calculation of the robot. The names of each joint and the actual range of degrees of freedom are shown in Table1.Table 1
List of joints.
Num.Joint nameFreedom range (°)1RShoulderPitch0∼1802RShoulderRoll0∼1803RElbow0∼1804LShoulderPitch0∼1805LShoulderRoll0∼1806LElbow0∼1807RHipRoll0∼1808RHipPitch30∼1809RKnee0∼15010RAnklePitch90∼18011RAnkleRoll70∼18012LHipRoll0∼18013LHipPitch0∼15014LKnee30∼18015LAnklePitch0∼18016LAnkleRoll0∼110In this paper, for each joint of the robot, its angle value is represented byθm, and mm=1,2,…,16 is the number of the joint. The nn=1,2,…,15 th key point captured by the motion capture is represented by point pn, and the vector formed by points pn1 and pn2 is pn1pn2. The key point numbers detected are shown in Figure 3.Figure 3
Correspondence of feature points.In the angle calculation, there are mainly three situations: the angle between the vector and the vector, the angle between the vector and the plane, and the angle between the plane and the plane [9]. The following examples illustrate the calculation formulas for the three cases.


#### 2.3.1. Calculation of the Angle between the Vector and the Vector

Here we takeθ3 as an example to introduce how to calculate Relbow. The value of θ3 is the angle between the vector p4p5 and the vector p5p6. The calculation formula is(2)θ3=p4p5,p5p6=arccosp4p5⋅p5p6p4p5p5p6.


#### 2.3.2. Calculation of the Angle between the Vector and the Plane

For example, the angleθ8 of the joint RHipPitch is the angle between vector p10p11 and the plane formed by vectors p3p10 and p3p2. To solve θ8, first use the cross product of p3p10 and p3p2 to obtain the normal vector n2,3,10 of the plane, then calculate the angle of the normal vector p10p11, and then calculate the remaining angles to obtain θ8:(3)θ8=π2−n2,3,10,p10p11=π2−arccosn2,3,10⋅p10p11n2,3,10p10p11.


#### 2.3.3. Calculation of the Angle between the Plane and the Plane

For example, if the RShoulderPitch angleθ1 of the joint is solved, the angle between the plane and the plane is solved. Specifically, first cross-multiply vectors p5p6 and p5p4 to obtain a plane normal vector n4,5,6, then cross-multiply vectors p2p4 and p2p3 to obtain a normal vector n2,3,4 of another plane, and then calculate the angle between the two normal vectors to obtain θ1:(4)θ1=n4,5,6,n2,3,4=arccosn4,5,6⋅n2,3,4n4,5,6n2,3,4.In the calculation process of each angle, due to the selection of the vector direction, it is necessary to find the complementary angle or the complementary angle according to the situation; considering the angle range of each joint, it is necessary to limit the calculated angle value to ensure that it is not exceeding the maximum allowable angle value.The above calculation process can get all the angle values except the two degrees of freedom of the ankle [10]. For the calculation of the angle of the ankle joint, the relative position of the robot and the ground needs to be taken into consideration because the sole of the foot must be parallel to the ground. As shown in Figure 4, θ10 is the angle between the vector p12p11 and the plane Zp10Y, and θ11 is the angle between the plane where the legs are located and the plane Zp10Y, where the plane and the legs are located is the plane formed by the vectors p12p11 and p11p10. θ15 and θ16 can be obtained in the same way.Figure 4
Schematic diagram of ankle joint angle calculation.


### 2.4. Action Selection Algorithm Based on Continuity

Just as a classification model cannot guarantee that the classification accuracy rate reaches 100%, the actions generated by the action generation model also guarantee that every frame generated is a real, coherent, and high-quality action. Therefore, in order to make the generated action suitable for music-based dance choreography, it is necessary to perform a coherence-based action screening first to remove mutation data, so as to ensure that the action data of each frame in an action segment is coherent and improve the quality of the generated action [11].If an action sequence is coherent, then the actions of adjacent frames in the sequence should have sufficient similarity, which is reflected in the small distance between the corresponding joint points. The motion data used in this article are sampled at a frequency of 30 frames per second. The distance between the corresponding joint point positions in two adjacent frames can be approximated as the speed of the joint at that moment. Because the overall speed of different action sequences is different, it is not easy to judge whether there is a sudden change or a pure rapid movement based solely on the absolute position change [12]. This study believes that, for a coherent action sequence, regardless of the overall speed, the rate of change of the speed between adjacent frames should be small, that is to say, for a sequence with a faster action speed, the instantaneous speed of each frame is higher. For a sequence with a slower action speed, the instantaneous speed of each frame is smaller, but for a continuous action sequence, the speed difference between adjacent frames should be relatively stable.Therefore, this paper conducts continuous screening of the action sequence based on the rate of speed change. First, calculate the sum of the absolute value of the first-order difference of the joint speeds of adjacent frames:(5)Vf=∑k=1cvf+1,k−vf,k,vf,k=xf+1k−xfk.Among them,f is the sequence number of the frame in the action segment Ni, x is the action vector, xfk is the k th dimensional action data of the f th frame, and c is the vector dimension of the action per frame. vf,k represents the speed of the kth dimensional data of the fth frame.Set the maximum thresholdεmax if the fth frame action satisfies Vf≤εmax; the action at that moment meets the continuity requirement. According to the condition, the action sequence is divided, and the continuous action frame that meets the condition is divided into a new action sequence. In order to remove new sequences whose length is too short after segmentation, a minimum length threshold value needs to be set, and only new sequences whose length exceeds this threshold value can be saved in the generated action database.


### 2.5. Choreography and Composition

Through the feature extraction and matching algorithm of music and action in the previous section, multiple action segments that match the target music have been obtained, and the connectivity constraints between adjacent segments have been met. In this section, on this basis, the adjacent action segments will be transitionally connected to solve the problem of sudden action changes. The action segments will be spliced into a complete sequence of actions to complete the final choreography. The sudden change of action referred to in this section refers to the fact that there is a certain distance between the action at the connection between the action segment and the adjacent action segmentNi+1, which causes the directly connected actions to be incoherent, which affects the visual effect of the dance.This section uses the intermediate frame interpolation algorithm to interpolate between the endk frames of the action segment M and the intermediate action of the action segment N according to the interpolation weight to obtain the final interpolation action. The interpolation action obtained by using this algorithm can realize the natural connection of two action segments and, at the same time, retain the characteristics of the ending action of the previous action segment to a certain extent [13]. The interpolation action ensures the continuity of the action, but there may be unreal actions such as footsteps. In order to avoid the long duration of the interpolation action and affect the perception, the value of k should not be too large. In this study, k = 14 is used in the experiment.Suppose the length of the action segmentM is m, the last k frame action is recorded as Mm−k+1=fm−k+1M,fm−k+2M,…,fmM, and the first k frame intermediate action of the action segment N is recorded as N1,…,k=f1N,f2N,…,fkN. First, the starting position of the intermediate action N1,...,k is translated to the same as Mm−k+1, and then, the interpolation action P1,…,k is synthesized. Among them, P1,…,k=f1p,f2p,…,fkp performs linear interpolation on the node displacement:(6)piP,s=αifm−k+iM,s+1−αifiN,s,1≤i≤k,αi=2⋅ik3−3⋅ik2+1,0≤i≤k−1.Among them,piP,s represents the coordinates of the sth joint point of the ith frame of the P action segment, and αi is the interpolation weight.


## 2.1. Overview

First of all, the problem that needs to be solved is motion capture based on Kinect. When Kinect captures human body motion data, it captures 20 joint nodes of the human body through infrared rays. As long as you face Kinect and do dance movements, you can record the corresponding raw data. This avoids the need to place sensors on the capturer during traditional motion capture, which hinders the fluency of dance movements. Convert the captured raw data into the BVH file format to save the dance moves. The motion data file in BVH format is one of the standard file formats in the motion capture industry and is widely supported by mainstream animation production software. When using Kinect to collect dance movements, it is conducive to the optimal display and file storage of virtual dances based on the “Principle of Optimal Movements” and “Principle of Minimal Data Simplification” [7].Secondly, it is necessary to edit and integrate dance movements. Using the 3D software MAYA as the platform, the basic dance moves stored in the BVH format are imported into MAYA for editing. The choreographer can make modifications and combinations based on the existing basic dance moves, such as modifying the angle of the limbs, the speed of a certain movement, and the number of repetitions [8]; at the same time, they can also arrange and integrate the sequence of specific dance moves to express different dance themes. In addition, the choreography of dance emphasizes innovation. Using Kinect equipment, you can capture newly designed dance moves at any time. After converting it into BVH format, it becomes the new basic action element.Finally, it is necessary to match the skeleton and the character and realize the three-dimensional display. After the preliminary editing of a certain dance movement is completed, the 3D characters can be driven to dance. MAYA provides the function of binding the skeleton to the model so that the 3D model follows the movement of the skeleton. Modify the bound model by adjusting the weight of the model, the size of the skeleton, and other factors so that it can express the dance effect as realistically as possible. For the choreography of large-scale dances, after the choreography of a certain model is completed in advance, the skinned models can be copied and arranged directly to create the ideal formation and stage effect.


## 2.2. Motion Capture

### 2.2.1. Two-Dimensional Key Point Recognition Network

The key points of the two-dimensional human body are identified using the Stacked Hourglass Network architecture. This network is constructed by loop nesting of the Hourglass subnetwork. The Hourglass subnetwork is composed of a basic structure called residual module. The residual module structure is shown in Figure1(a). The input value will pass through the upper and lower paths. This design is inspired by the residual network. The above path is a convolution path, which contains three convolution layers with different sizes of convolution kernels, which are represented by white rectangles in the figure. The three lines of text in the rectangle represent the number of input channels (NIn), the size of the convolution kernel (K), and the number of output channels (NOut) from top to bottom. Among the three convolutional layers, there are also batch regularization layers and nonlinear activation layers (RELUs), all of which are represented by gray rectangles. The lower path is a jumper path. This path contains only a convolutional layer with a convolution kernel size of 1 × 1, which will change the number of channels of the input value, and its output value will be directly added to the output value of the upper path.Figure 1
Two-dimensional key point recognition network. (a) Residual module structure. (b)Hourglass subnetwork structure.
(a)(b)The structure of the Hourglass subnetwork is shown in Figure1(b). Each white rectangle in the figure represents one of the above residual modules. In the upper half, feature extraction is performed continuously, and in the lower half, downsampling is performed through the maximum pooling operation first, and after several residual modules, the nearest neighbor interpolation method is used for upsampling. The two gray rectangles in the figure represent the downsampling and upsampling processes, respectively. It can be seen that there is a dashed line frame in the Hourglass subnet structure diagram. The dashed line frame is replaced by an Hourglass subnet, which is called a second-order Hourglass network. In this study, the Hourglass subnetwork is looped and nested four times for two-dimensional key point detection, namely, the fourth-order Hourglass network.


### 2.2.2. Three-Dimensional Keypoint Regression Neural Network

After the two-dimensional human body key points are recognized, a set of two-dimensional coordinate values of the human body key points will be output. Then, use this set of two-dimensional coordinatesx∈R2n and use the neural network to return to the corresponding set of three-dimensional coordinate values y∈R3n. The mapping relationship of the three-dimensional coordinates regressed from the two-dimensional coordinates is f:R2n⟶R3n. Optimize the neural network by minimizing the following prediction errors, namely,(1)f∗=min1N∑i=1NLfxi−yi.Among them,L represents the Euclidean distance between vectors as the loss function, N represents the total number of key points to be identified, and f∗ represents a regression neural network. The structure of the regression neural network model is shown in Figure 2. First, the dimension is converted to 1024 dimensions through a fully connected layer. After that, it passes through the batch regularization layer, the RELUs layer, and the Dropout layer in turn and contains a jump-through path. The structure in the dashed frame in Figure 2 will be repeated once, and finally, a 3n-dimensional output vector will be formed through a fully connected layer. So far, the three-dimensional coordinates of the key points of the presenter are obtained.Figure 2
Three-dimensional key point regression neural network.


## 2.2.1. Two-Dimensional Key Point Recognition Network

The key points of the two-dimensional human body are identified using the Stacked Hourglass Network architecture. This network is constructed by loop nesting of the Hourglass subnetwork. The Hourglass subnetwork is composed of a basic structure called residual module. The residual module structure is shown in Figure1(a). The input value will pass through the upper and lower paths. This design is inspired by the residual network. The above path is a convolution path, which contains three convolution layers with different sizes of convolution kernels, which are represented by white rectangles in the figure. The three lines of text in the rectangle represent the number of input channels (NIn), the size of the convolution kernel (K), and the number of output channels (NOut) from top to bottom. Among the three convolutional layers, there are also batch regularization layers and nonlinear activation layers (RELUs), all of which are represented by gray rectangles. The lower path is a jumper path. This path contains only a convolutional layer with a convolution kernel size of 1 × 1, which will change the number of channels of the input value, and its output value will be directly added to the output value of the upper path.Figure 1
Two-dimensional key point recognition network. (a) Residual module structure. (b)Hourglass subnetwork structure.
(a)(b)The structure of the Hourglass subnetwork is shown in Figure1(b). Each white rectangle in the figure represents one of the above residual modules. In the upper half, feature extraction is performed continuously, and in the lower half, downsampling is performed through the maximum pooling operation first, and after several residual modules, the nearest neighbor interpolation method is used for upsampling. The two gray rectangles in the figure represent the downsampling and upsampling processes, respectively. It can be seen that there is a dashed line frame in the Hourglass subnet structure diagram. The dashed line frame is replaced by an Hourglass subnet, which is called a second-order Hourglass network. In this study, the Hourglass subnetwork is looped and nested four times for two-dimensional key point detection, namely, the fourth-order Hourglass network.


## 2.2.2. Three-Dimensional Keypoint Regression Neural Network

After the two-dimensional human body key points are recognized, a set of two-dimensional coordinate values of the human body key points will be output. Then, use this set of two-dimensional coordinatesx∈R2n and use the neural network to return to the corresponding set of three-dimensional coordinate values y∈R3n. The mapping relationship of the three-dimensional coordinates regressed from the two-dimensional coordinates is f:R2n⟶R3n. Optimize the neural network by minimizing the following prediction errors, namely,(1)f∗=min1N∑i=1NLfxi−yi.Among them,L represents the Euclidean distance between vectors as the loss function, N represents the total number of key points to be identified, and f∗ represents a regression neural network. The structure of the regression neural network model is shown in Figure 2. First, the dimension is converted to 1024 dimensions through a fully connected layer. After that, it passes through the batch regularization layer, the RELUs layer, and the Dropout layer in turn and contains a jump-through path. The structure in the dashed frame in Figure 2 will be repeated once, and finally, a 3n-dimensional output vector will be formed through a fully connected layer. So far, the three-dimensional coordinates of the key points of the presenter are obtained.Figure 2
Three-dimensional key point regression neural network.


## 2.3. Action Recurrence

After obtaining the coordinates of the key points of the presenter, they need to be mapped to the robot. The control of the robot takes the angle of each joint as the control command, and the process of using the coordinate value of each key point to calculate the joint angle is called inverse kinematics calculation of the robot. The names of each joint and the actual range of degrees of freedom are shown in Table1.Table 1
List of joints.
Num.Joint nameFreedom range (°)1RShoulderPitch0∼1802RShoulderRoll0∼1803RElbow0∼1804LShoulderPitch0∼1805LShoulderRoll0∼1806LElbow0∼1807RHipRoll0∼1808RHipPitch30∼1809RKnee0∼15010RAnklePitch90∼18011RAnkleRoll70∼18012LHipRoll0∼18013LHipPitch0∼15014LKnee30∼18015LAnklePitch0∼18016LAnkleRoll0∼110In this paper, for each joint of the robot, its angle value is represented byθm, and mm=1,2,…,16 is the number of the joint. The nn=1,2,…,15 th key point captured by the motion capture is represented by point pn, and the vector formed by points pn1 and pn2 is pn1pn2. The key point numbers detected are shown in Figure 3.Figure 3
Correspondence of feature points.In the angle calculation, there are mainly three situations: the angle between the vector and the vector, the angle between the vector and the plane, and the angle between the plane and the plane [9]. The following examples illustrate the calculation formulas for the three cases.


### 2.3.1. Calculation of the Angle between the Vector and the Vector

Here we takeθ3 as an example to introduce how to calculate Relbow. The value of θ3 is the angle between the vector p4p5 and the vector p5p6. The calculation formula is(2)θ3=p4p5,p5p6=arccosp4p5⋅p5p6p4p5p5p6.


### 2.3.2. Calculation of the Angle between the Vector and the Plane

For example, the angleθ8 of the joint RHipPitch is the angle between vector p10p11 and the plane formed by vectors p3p10 and p3p2. To solve θ8, first use the cross product of p3p10 and p3p2 to obtain the normal vector n2,3,10 of the plane, then calculate the angle of the normal vector p10p11, and then calculate the remaining angles to obtain θ8:(3)θ8=π2−n2,3,10,p10p11=π2−arccosn2,3,10⋅p10p11n2,3,10p10p11.


### 2.3.3. Calculation of the Angle between the Plane and the Plane

For example, if the RShoulderPitch angleθ1 of the joint is solved, the angle between the plane and the plane is solved. Specifically, first cross-multiply vectors p5p6 and p5p4 to obtain a plane normal vector n4,5,6, then cross-multiply vectors p2p4 and p2p3 to obtain a normal vector n2,3,4 of another plane, and then calculate the angle between the two normal vectors to obtain θ1:(4)θ1=n4,5,6,n2,3,4=arccosn4,5,6⋅n2,3,4n4,5,6n2,3,4.In the calculation process of each angle, due to the selection of the vector direction, it is necessary to find the complementary angle or the complementary angle according to the situation; considering the angle range of each joint, it is necessary to limit the calculated angle value to ensure that it is not exceeding the maximum allowable angle value.The above calculation process can get all the angle values except the two degrees of freedom of the ankle [10]. For the calculation of the angle of the ankle joint, the relative position of the robot and the ground needs to be taken into consideration because the sole of the foot must be parallel to the ground. As shown in Figure 4, θ10 is the angle between the vector p12p11 and the plane Zp10Y, and θ11 is the angle between the plane where the legs are located and the plane Zp10Y, where the plane and the legs are located is the plane formed by the vectors p12p11 and p11p10. θ15 and θ16 can be obtained in the same way.Figure 4
Schematic diagram of ankle joint angle calculation.


## 2.3.1. Calculation of the Angle between the Vector and the Vector

Here we takeθ3 as an example to introduce how to calculate Relbow. The value of θ3 is the angle between the vector p4p5 and the vector p5p6. The calculation formula is(2)θ3=p4p5,p5p6=arccosp4p5⋅p5p6p4p5p5p6.


## 2.3.2. Calculation of the Angle between the Vector and the Plane

For example, the angleθ8 of the joint RHipPitch is the angle between vector p10p11 and the plane formed by vectors p3p10 and p3p2. To solve θ8, first use the cross product of p3p10 and p3p2 to obtain the normal vector n2,3,10 of the plane, then calculate the angle of the normal vector p10p11, and then calculate the remaining angles to obtain θ8:(3)θ8=π2−n2,3,10,p10p11=π2−arccosn2,3,10⋅p10p11n2,3,10p10p11.


## 2.3.3. Calculation of the Angle between the Plane and the Plane

For example, if the RShoulderPitch angleθ1 of the joint is solved, the angle between the plane and the plane is solved. Specifically, first cross-multiply vectors p5p6 and p5p4 to obtain a plane normal vector n4,5,6, then cross-multiply vectors p2p4 and p2p3 to obtain a normal vector n2,3,4 of another plane, and then calculate the angle between the two normal vectors to obtain θ1:(4)θ1=n4,5,6,n2,3,4=arccosn4,5,6⋅n2,3,4n4,5,6n2,3,4.In the calculation process of each angle, due to the selection of the vector direction, it is necessary to find the complementary angle or the complementary angle according to the situation; considering the angle range of each joint, it is necessary to limit the calculated angle value to ensure that it is not exceeding the maximum allowable angle value.The above calculation process can get all the angle values except the two degrees of freedom of the ankle [10]. For the calculation of the angle of the ankle joint, the relative position of the robot and the ground needs to be taken into consideration because the sole of the foot must be parallel to the ground. As shown in Figure 4, θ10 is the angle between the vector p12p11 and the plane Zp10Y, and θ11 is the angle between the plane where the legs are located and the plane Zp10Y, where the plane and the legs are located is the plane formed by the vectors p12p11 and p11p10. θ15 and θ16 can be obtained in the same way.Figure 4
Schematic diagram of ankle joint angle calculation.


## 2.4. Action Selection Algorithm Based on Continuity

Just as a classification model cannot guarantee that the classification accuracy rate reaches 100%, the actions generated by the action generation model also guarantee that every frame generated is a real, coherent, and high-quality action. Therefore, in order to make the generated action suitable for music-based dance choreography, it is necessary to perform a coherence-based action screening first to remove mutation data, so as to ensure that the action data of each frame in an action segment is coherent and improve the quality of the generated action [11].If an action sequence is coherent, then the actions of adjacent frames in the sequence should have sufficient similarity, which is reflected in the small distance between the corresponding joint points. The motion data used in this article are sampled at a frequency of 30 frames per second. The distance between the corresponding joint point positions in two adjacent frames can be approximated as the speed of the joint at that moment. Because the overall speed of different action sequences is different, it is not easy to judge whether there is a sudden change or a pure rapid movement based solely on the absolute position change [12]. This study believes that, for a coherent action sequence, regardless of the overall speed, the rate of change of the speed between adjacent frames should be small, that is to say, for a sequence with a faster action speed, the instantaneous speed of each frame is higher. For a sequence with a slower action speed, the instantaneous speed of each frame is smaller, but for a continuous action sequence, the speed difference between adjacent frames should be relatively stable.Therefore, this paper conducts continuous screening of the action sequence based on the rate of speed change. First, calculate the sum of the absolute value of the first-order difference of the joint speeds of adjacent frames:(5)Vf=∑k=1cvf+1,k−vf,k,vf,k=xf+1k−xfk.Among them,f is the sequence number of the frame in the action segment Ni, x is the action vector, xfk is the k th dimensional action data of the f th frame, and c is the vector dimension of the action per frame. vf,k represents the speed of the kth dimensional data of the fth frame.Set the maximum thresholdεmax if the fth frame action satisfies Vf≤εmax; the action at that moment meets the continuity requirement. According to the condition, the action sequence is divided, and the continuous action frame that meets the condition is divided into a new action sequence. In order to remove new sequences whose length is too short after segmentation, a minimum length threshold value needs to be set, and only new sequences whose length exceeds this threshold value can be saved in the generated action database.


## 2.5. Choreography and Composition

Through the feature extraction and matching algorithm of music and action in the previous section, multiple action segments that match the target music have been obtained, and the connectivity constraints between adjacent segments have been met. In this section, on this basis, the adjacent action segments will be transitionally connected to solve the problem of sudden action changes. The action segments will be spliced into a complete sequence of actions to complete the final choreography. The sudden change of action referred to in this section refers to the fact that there is a certain distance between the action at the connection between the action segment and the adjacent action segmentNi+1, which causes the directly connected actions to be incoherent, which affects the visual effect of the dance.This section uses the intermediate frame interpolation algorithm to interpolate between the endk frames of the action segment M and the intermediate action of the action segment N according to the interpolation weight to obtain the final interpolation action. The interpolation action obtained by using this algorithm can realize the natural connection of two action segments and, at the same time, retain the characteristics of the ending action of the previous action segment to a certain extent [13]. The interpolation action ensures the continuity of the action, but there may be unreal actions such as footsteps. In order to avoid the long duration of the interpolation action and affect the perception, the value of k should not be too large. In this study, k = 14 is used in the experiment.Suppose the length of the action segmentM is m, the last k frame action is recorded as Mm−k+1=fm−k+1M,fm−k+2M,…,fmM, and the first k frame intermediate action of the action segment N is recorded as N1,…,k=f1N,f2N,…,fkN. First, the starting position of the intermediate action N1,...,k is translated to the same as Mm−k+1, and then, the interpolation action P1,…,k is synthesized. Among them, P1,…,k=f1p,f2p,…,fkp performs linear interpolation on the node displacement:(6)piP,s=αifm−k+iM,s+1−αifiN,s,1≤i≤k,αi=2⋅ik3−3⋅ik2+1,0≤i≤k−1.Among them,piP,s represents the coordinates of the sth joint point of the ith frame of the P action segment, and αi is the interpolation weight.


## 3. Experiments and Results

### 3.1. Dataset

In recent years, the research on dance movement synthesis is mainly based on motion capture data. At present, the main public motion capture motion datasets are as follows. (1) SBU dataset includes 8 types of actions, a total of 230 sequences of 6614 frames, but the actions in the dataset are all nondance movements such as handshake and punching. (2) HDM05 data set contains about 100 action types, with a total of 2337 sequences of 1840046 frames, but basically all kinds of nondance actions such as walking and kicking. (3) UCY dataset (University of Cyprus), containing a total of 161 sequences 147509 frames, providing dance moves in Greek, Cypriot dance, and other styles, but only 8 of them are relatively complete movements accompanied by music, a total of 28892 frames, and the rest of the sequences are single movement fragments with a short time, which is not conducive to complete dance [14]. (4) The CMU dataset contains a total of 2235 sequences of 98,7341 frames. This is the largest motion capture dataset published so far, covering a wide range of motion types, of which only 64,300 frames are for pure dance movements, with no accompanying music. To sum up, although there are some public exercise datasets, most of them are not dance moves, and there are very few dance moves’ data accompanied by music [15]. For the study of the relationship between music and dance movements based on deep learning, movement and music data play a key role in the training of the model.Therefore, this paper additionally constructs a music-action dataset composed of a complete music choreography sequence. Compared with using professional dancers, it is more economical and convenient to obtain enough dance data through motion capture equipment and download the motion data in VMD format corresponding to different music on the Internet. This paper uses the VMD action files obtained from the Internet and the accompanying WAV music files to construct a music-dance action dataset, with a total of 192 segments, 1057344 frames, and about 587 minutes. Each segment is an independent dance.The music and dance styles contained in the dataset constructed in this study are not exactly the same, and the speed is also fast and slow. Therefore, it is necessary to classify the actions before performing network training. Based on manual experience, combined with the overall characteristics of the 192 songs and dances in the dataset, the overall style of music and dance can be divided into three categories: modern dance, street dance, and house dance. The overall speed is divided into two categories, fast and slow, for a total of six categories. Since the overall speed of the movement in the same song and dance is not static, for example, the movement is more relaxed at the beginning and end of the song and the movement is more intense at the climax of the song, so the overall dance speed of the entire song is not enough. To classify actions, based on the overall speed of the dance, this study continues to divide movements into fast and slow based on the frame granularity. Based on the above principles, this study manually annotates the constructed dataset. Table2 shows the number of frames and duration of various actions after classification.Table 2
Detailed information of various types of data in the dataset.
Dance styleOverall speedNumber of fragmentsFrame rateNumber of framesDuration (min)House danceFast54Fast285393158.6Slow2642814.7Slow35Fast15717587.3Slow5403430.0Street danceFast67Fast357287197.4Slow1804810.0Slow6Fast2550413.7Slow90634.9Modern danceFast5Fast2167912.6Slow2490.1Slow25Fast4151622.6Slow5641031.9


### 3.2. Model Training and Prediction

In order to obtain good results in deep neural network training, it is necessary to provide sufficient data so that the neural network can fully tap the inner relationship between the data. For the three types of dances, house dance, street dance, and modern dance are included in the dataset constructed in this paper; according to the data volume of different speed movements, this paper trains different movement generation models. During training, 12 Gaussian distributions are used to form a mixed model (m = 12), the number of batches (batch size) is set to 100, the sequence length of one-time input is set to 120, the learning rate is 500, and the total training is 500 cycles (epoch), optimized using the RMS Prop optimizer and the learning rate is set to 0.01. Table 3 shows the training results at different times.Table 3
Action generation results at different times (street dance actions).
EpochTimeGenerate sample action posesDescribe12.5 minUnable to generate qualified actions, most joint positions are random, and the outline of the human body can be vaguely seen50∼2 hUnderstand the joint positions and be able to generate basic movements200∼9 hThe relative relationship of joints is more stable, which can generate richer and more stable movements500∼22 hThe generated action has unstable continuity and may be overfittingFigure5 shows the model loss of the training set and validation set during model training. It is worth noting that the error function Eq needs to be minimized during the model training process. Unlike the commonly used loss functions in other networks (such as cross entropy), Eq does not meet the condition of constant greater than zero. Therefore, when the model loss is less than zero, the training process will be terminated in advance, as shown in Figure 5(a). The loss of the validation set and the loss of the training set are inconsistent, and there is even no obvious downward trend. This is because dance movement generation tasks are different from other tasks such as target classification. The choreography and expression of dance moves are not unique. This is where the diversity of dance moves lies. The training process of the movement generation model seeks regularity in it. The diversity and regularity of data are a pair of contradictory standards. Although the actions in the dataset have been preliminarily classified, in the limited dataset, it is difficult to ensure that the verification set selected at random each time has the same law as the training set. Therefore, in the verification set during the first 350 cycles of training, the loss has not changed much. As the model continues to be trained, overfitting occurs, resulting in a sharp increase in the loss of the validation set. You can refer to the loss of the validation set to judge the training situation of the model.Figure 5
Model loss. (a) Training set. (b) Validation set.
(a)(b)


### 3.3. Qualitative Results

According to the visual effect of the synthesized dance, the choreography result is evaluated, and the effect of the algorithm in this paper is evaluated. First, extract the overall characteristics of “Tokyo Teddy Bear;” after calculation, the BPM value is 126.05, the duration of the change note is 1.93, and the rapid house dance motion generation model is selected to generate the candidate motion database, which is consistent with the user’s intuitive hearing. Observing the final dance effect, you can feel that the rhythm and intensity of the dance and the target music match to a certain extent, and the movements are smooth and coherent. Figure6 shows a posture snapshot of the synthesized dance. Only from the intuitive visual effects, the choreography algorithm in this paper can be considered to be effective.Figure 6
“Tokyo teddy bear” synthetic dance pose snapshot.


### 3.4. Quantitative Results

Experimental purpose: use user ratings to analyze the synthesizing effects of three styles of dance. First, analyze the style of the music according to the overall characteristics of the target music (BPM and the duration of changing notes) and generate the corresponding choreography. This experiment analyzes a number of target music, selects three target music suitable for generating hip-hop, house dance, and modern dance, and choreographs them. The detailed information and overall characteristic values of the target music are shown in Table4. The test users, respectively, judged the dance styles of the three segments and evaluated the matching degree of music and dance and the continuity and authenticity of dance movements, and the results are shown in Table 5.Table 4
Details of the three target music.
Music nameDurationMusic BPMVarying note durationDance styleBad apple00:05:10138.61.57Street danceChocolate00:03:46129.21.66House danceFrom Y to Y00:05:3096.42.54Modern danceTable 5
Manual scoring of three styles of dance.
Dance styleMusic and dance matchContinuityAuthenticityStyle judgment accuracy (%)Street dance4.28 ± 0.104.07 ± 0.074.04 ± 0.1096.9House dance4.13 ± 0.094.05 ± 0.084.02 ± 0.0896.9Modern dance4.22 ± 0.114.08 ± 0.084.08 ± 0.11100From the results, it can be found that the average value of the coherence, authenticity, and matching with music of the three styles of dance are all above 4 points, indicating that users are satisfied with the results of the synthesized dance styles. Therefore, the music choreography algorithm proposed in this paper is Effective. The three indicators of street dance have the highest scores, followed by modern dance, and finally house dance. This study interviewed these users after they completed the test. The users said that the rhythm of hip-hop style movements is obvious and the range of movements is larger, while the range of house dance movements is generally small. Sometimes, it is difficult for users to distinguish between small movements and jitter data. After analysis, the reason why the synthesis result of house dance is worse than that of street dance is that, in the action dataset constructed in this study, the diversity of house dance actions is more abundant, and the concentration of actions is worse than that of street dance, so it is not conducive to the training and learning of the action generation model. Among the 33 users who participated in the score, only 1 user made a wrong judgment on the style of hip-hop and home dance, and the judgments of other users were accurate. After follow-up interviews, the user stated that it was because she did not know the specific concepts of home dance and hip-hop and unable to judge.


## 3.1. Dataset

In recent years, the research on dance movement synthesis is mainly based on motion capture data. At present, the main public motion capture motion datasets are as follows. (1) SBU dataset includes 8 types of actions, a total of 230 sequences of 6614 frames, but the actions in the dataset are all nondance movements such as handshake and punching. (2) HDM05 data set contains about 100 action types, with a total of 2337 sequences of 1840046 frames, but basically all kinds of nondance actions such as walking and kicking. (3) UCY dataset (University of Cyprus), containing a total of 161 sequences 147509 frames, providing dance moves in Greek, Cypriot dance, and other styles, but only 8 of them are relatively complete movements accompanied by music, a total of 28892 frames, and the rest of the sequences are single movement fragments with a short time, which is not conducive to complete dance [14]. (4) The CMU dataset contains a total of 2235 sequences of 98,7341 frames. This is the largest motion capture dataset published so far, covering a wide range of motion types, of which only 64,300 frames are for pure dance movements, with no accompanying music. To sum up, although there are some public exercise datasets, most of them are not dance moves, and there are very few dance moves’ data accompanied by music [15]. For the study of the relationship between music and dance movements based on deep learning, movement and music data play a key role in the training of the model.Therefore, this paper additionally constructs a music-action dataset composed of a complete music choreography sequence. Compared with using professional dancers, it is more economical and convenient to obtain enough dance data through motion capture equipment and download the motion data in VMD format corresponding to different music on the Internet. This paper uses the VMD action files obtained from the Internet and the accompanying WAV music files to construct a music-dance action dataset, with a total of 192 segments, 1057344 frames, and about 587 minutes. Each segment is an independent dance.The music and dance styles contained in the dataset constructed in this study are not exactly the same, and the speed is also fast and slow. Therefore, it is necessary to classify the actions before performing network training. Based on manual experience, combined with the overall characteristics of the 192 songs and dances in the dataset, the overall style of music and dance can be divided into three categories: modern dance, street dance, and house dance. The overall speed is divided into two categories, fast and slow, for a total of six categories. Since the overall speed of the movement in the same song and dance is not static, for example, the movement is more relaxed at the beginning and end of the song and the movement is more intense at the climax of the song, so the overall dance speed of the entire song is not enough. To classify actions, based on the overall speed of the dance, this study continues to divide movements into fast and slow based on the frame granularity. Based on the above principles, this study manually annotates the constructed dataset. Table2 shows the number of frames and duration of various actions after classification.Table 2
Detailed information of various types of data in the dataset.
Dance styleOverall speedNumber of fragmentsFrame rateNumber of framesDuration (min)House danceFast54Fast285393158.6Slow2642814.7Slow35Fast15717587.3Slow5403430.0Street danceFast67Fast357287197.4Slow1804810.0Slow6Fast2550413.7Slow90634.9Modern danceFast5Fast2167912.6Slow2490.1Slow25Fast4151622.6Slow5641031.9


## 3.2. Model Training and Prediction

In order to obtain good results in deep neural network training, it is necessary to provide sufficient data so that the neural network can fully tap the inner relationship between the data. For the three types of dances, house dance, street dance, and modern dance are included in the dataset constructed in this paper; according to the data volume of different speed movements, this paper trains different movement generation models. During training, 12 Gaussian distributions are used to form a mixed model (m = 12), the number of batches (batch size) is set to 100, the sequence length of one-time input is set to 120, the learning rate is 500, and the total training is 500 cycles (epoch), optimized using the RMS Prop optimizer and the learning rate is set to 0.01. Table 3 shows the training results at different times.Table 3
Action generation results at different times (street dance actions).
EpochTimeGenerate sample action posesDescribe12.5 minUnable to generate qualified actions, most joint positions are random, and the outline of the human body can be vaguely seen50∼2 hUnderstand the joint positions and be able to generate basic movements200∼9 hThe relative relationship of joints is more stable, which can generate richer and more stable movements500∼22 hThe generated action has unstable continuity and may be overfittingFigure5 shows the model loss of the training set and validation set during model training. It is worth noting that the error function Eq needs to be minimized during the model training process. Unlike the commonly used loss functions in other networks (such as cross entropy), Eq does not meet the condition of constant greater than zero. Therefore, when the model loss is less than zero, the training process will be terminated in advance, as shown in Figure 5(a). The loss of the validation set and the loss of the training set are inconsistent, and there is even no obvious downward trend. This is because dance movement generation tasks are different from other tasks such as target classification. The choreography and expression of dance moves are not unique. This is where the diversity of dance moves lies. The training process of the movement generation model seeks regularity in it. The diversity and regularity of data are a pair of contradictory standards. Although the actions in the dataset have been preliminarily classified, in the limited dataset, it is difficult to ensure that the verification set selected at random each time has the same law as the training set. Therefore, in the verification set during the first 350 cycles of training, the loss has not changed much. As the model continues to be trained, overfitting occurs, resulting in a sharp increase in the loss of the validation set. You can refer to the loss of the validation set to judge the training situation of the model.Figure 5
Model loss. (a) Training set. (b) Validation set.
(a)(b)


## 3.3. Qualitative Results

According to the visual effect of the synthesized dance, the choreography result is evaluated, and the effect of the algorithm in this paper is evaluated. First, extract the overall characteristics of “Tokyo Teddy Bear;” after calculation, the BPM value is 126.05, the duration of the change note is 1.93, and the rapid house dance motion generation model is selected to generate the candidate motion database, which is consistent with the user’s intuitive hearing. Observing the final dance effect, you can feel that the rhythm and intensity of the dance and the target music match to a certain extent, and the movements are smooth and coherent. Figure6 shows a posture snapshot of the synthesized dance. Only from the intuitive visual effects, the choreography algorithm in this paper can be considered to be effective.Figure 6
“Tokyo teddy bear” synthetic dance pose snapshot.


## 3.4. Quantitative Results

Experimental purpose: use user ratings to analyze the synthesizing effects of three styles of dance. First, analyze the style of the music according to the overall characteristics of the target music (BPM and the duration of changing notes) and generate the corresponding choreography. This experiment analyzes a number of target music, selects three target music suitable for generating hip-hop, house dance, and modern dance, and choreographs them. The detailed information and overall characteristic values of the target music are shown in Table4. The test users, respectively, judged the dance styles of the three segments and evaluated the matching degree of music and dance and the continuity and authenticity of dance movements, and the results are shown in Table 5.Table 4
Details of the three target music.
Music nameDurationMusic BPMVarying note durationDance styleBad apple00:05:10138.61.57Street danceChocolate00:03:46129.21.66House danceFrom Y to Y00:05:3096.42.54Modern danceTable 5
Manual scoring of three styles of dance.
Dance styleMusic and dance matchContinuityAuthenticityStyle judgment accuracy (%)Street dance4.28 ± 0.104.07 ± 0.074.04 ± 0.1096.9House dance4.13 ± 0.094.05 ± 0.084.02 ± 0.0896.9Modern dance4.22 ± 0.114.08 ± 0.084.08 ± 0.11100From the results, it can be found that the average value of the coherence, authenticity, and matching with music of the three styles of dance are all above 4 points, indicating that users are satisfied with the results of the synthesized dance styles. Therefore, the music choreography algorithm proposed in this paper is Effective. The three indicators of street dance have the highest scores, followed by modern dance, and finally house dance. This study interviewed these users after they completed the test. The users said that the rhythm of hip-hop style movements is obvious and the range of movements is larger, while the range of house dance movements is generally small. Sometimes, it is difficult for users to distinguish between small movements and jitter data. After analysis, the reason why the synthesis result of house dance is worse than that of street dance is that, in the action dataset constructed in this study, the diversity of house dance actions is more abundant, and the concentration of actions is worse than that of street dance, so it is not conducive to the training and learning of the action generation model. Among the 33 users who participated in the score, only 1 user made a wrong judgment on the style of hip-hop and home dance, and the judgments of other users were accurate. After follow-up interviews, the user stated that it was because she did not know the specific concepts of home dance and hip-hop and unable to judge.


## 4. Conclusion

This paper proposes a dance choreography algorithm based on the deep learning model, starting from improving the harmony of music and choreography. First, extract the overall characteristics of the music, including the average duration of BPM and notes, and perform preliminary matching of the action speed characteristics based on this. Then, complete segmentation of music and action sequences and feature extraction of rhythm intensity and combine the results of feature matching and connectability analysis to obtain action sequences that match the target music. Finally, the adjacent action fragments are interpolated to complete the computer music choreography. In this paper, qualitative and quantitative experiments are designed to evaluate the effectiveness of the algorithm. The experimental results show that the local bone movement speed feature extraction algorithm and the dance spatial feature extraction algorithm proposed in this paper can effectively reflect the corresponding characteristics of dance; the overall characteristics of music can accurately reflect the style of music, and the corresponding styles can be synthesized accordingly. Choreographed action: the hierarchical feature matching algorithm is better than the feature matching algorithm based on local rhythm and intensity. The addition of the overall feature matching can synthesize a dance that is more matched to the target music. In the future, it is necessary to further improve the accuracy of the model through the optimization of the model structure to make the generated actions more realistic.



---

*Source: 2352024-2022-01-19.xml*