# Dance Action Generation Model Based on Recurrent Neural Network

Authors: Xuan Ma; Kai Wang
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

In recent years, modeling methods based on Deep Learning and Recurrent Neural Network (RNN) have achieved rapid development in various tasks in the field of computer vision, and video generation technology has also achieved remarkable results on this basis, with boosting effect. This paper analyzes the requirements and system design of the dance generation system based on cyclic neural network and clearly puts forward the specific functional requirements and nonfunctional requirements. Design, test each function of the system, and show the graphs generated by our application during training and testing, and visualize the specific value of the loss function during training. Combining the latest image generation technology and the most effective open source gesture detection project, a dance generation algorithm is completed, which can generate a video of the target person completing the specified dance action.


---

## Body

## 1. Introduction

The traditional manual drawing of dance scores by observing human movements is very time-consuming. An ordinary dance of about 30 minutes takes a professional dance expert a month to complete. With the development and popularization of deep learning in recent years, artificial neural networks have been successfully applied to the generation of dance movements. A significant advantage of using deep learning for dance generation is their ability to directly extract high-level features from raw data (audio, motion capture data, etc.). In addition, deep neural networks are able to create new dance moves. Recurrent neural networks have been widely used to generate human face images and natural scene images, and there are currently many methods to stabilize the adversarial training process and significantly improve the generated image quality. In order to better solve the general video synthesis problem, we need to understand how the image pixels change in order to generate a complete object action picture with temporal dimension. The precise motion between frames of a moving object in a video has high uncertainty, but by simulating the uncertainty of the underlying structure of the moving object, the above problem becomes tractable. Assuming that our input data is human pose, from the application’s point of view, the user usually considers specific intentions such as changing the background, object's class, color, appearance information, etc. We can express this intent in an appropriate way to explicitly guide the generation process to achieve direct control over the image generation process. Specifically, we believe that it is reasonable to generate images by conditioning on a reference image and specifying poses. The reference image is used as the input condition of the network, and the model has enough information about the appearance of the expected object in advance, while the guidance given by the expected pose skeleton map is both explicit and flexible. So in principle, this method can manipulate any object to accomplish any pose. In this work, we focus on transferring a person from a given pose to a predetermined pose. Many interesting applications derive from this task. For example, in movie production, we can directly manipulate a character’s body into a desired pose; or for human pose estimation, we can generate training data for rare but important poses [1–8]. In this paper, a multilayer joint network model lower limb action recognition algorithm based on long short-term memory neural network (LSTM) and convolutional neural network (CONVNET) is proposed. The generation of lower limb movements in dance needs to consider the dynamic relationship of movements. The self-connected network structure of LSTM makes it have a natural advantage in processing dynamically changing time series data, and CONVNENT is very good at learning the spatial information of movements. The LSTM model is fused with the CNN model, so that the two can complement each other and achieve a more accurate recognition effect.


## 2. Related Works

With the advancement of machine learning technology, various machine learning algorithms have been proposed to generate dance moves in the form of motion capture data. Hidden Markov Models (HMMs) are also used to generate dance moves. As early as 2005, Yi Wang et al. proposed a new model NPHHMM (nonparameter hierarchy Hidden Markov Model), which is a hierarchical Hidden Markov Model with nonparametric output density. They applied it to create a motion engine for learning and reproducible creation of human movements. In their study, they trained NPHHMM on motion capture data containing ballet steps, ballet rolls, disco, and complex disco movements. In 2012, Tape et al., inspired by earlier papers, proposed a new framework for learning many-to-many statistical mappings from musical features to dance graphs. Based on these statistical maps, they defined a discrete Hidden Markov Model and synthesized candidate dance figure sequences using a modified Viterbi algorithm [9]. The model is able to classify music genres based on the Mel-Frequency Cepstrum Coefficients (MFCC) features of the music and generate corresponding dance moves based on the music genres. In 2016, Luka Crnkovic-Friis and Louise Crnkovic-Friis proposed a system called Chor-RNN to generate novel dance material with meticulous dance language and specific styles. It employs a mixed-density long-short-term memory (LSTM) type RNN and is trained with motion-captured modern dance. After 48 hours of training, the model was able to understand and generate choreography grammars and styles well and to some extent semantics. This is the first attempt to apply deep recurrent neural networks to the field of automatic choreography and successfully generate new choreographic sequences. However, the model only considers generating dance moves; it does not provide any way to accompany any music. Inspired by Crnkovic-Friis, in 2017 Omid Alemi et al. introduced an application called GrooveNet, which uses Conditional Restricted Boltzmann Machines (FCRBM) and recurrent neural networks. The network (RNN) generates dance moves. The results show that the application is able to learn and generate dance patterns from a very small training set. However, if the music used is not included in the training set, the effect is not ideal. In 2018, Juheon Lee et al. proposed a neural network-based model that can generate novel and natural dance moves. The researchers obtained audio-video data pairs from YouTube and used it as a training set. The model extracts the x, y coordinates from the video as motion features and then extracts Mel as audio features. The skeleton sequence is then generated using an autoregressive encoder-decoder network, and the results are compared with the real data using the MSE loss as a cost function. However, if the training dataset is not large enough, the system is not stable enough. A well-designed deep recurrent neural network is proposed by Nelson Yalta et al. The model employs one-dimensional convolutional layers and multiple long short-term memory (LSTM) layers as the encoder part to process the audio power spectrum and then utilizes the LSTM layer as the decoder part to generate dance movements. In the decoder part, the self-encoder module is used to reduce the accumulation of feedback errors. The model has reliable motion generation performance and can be applied in real time. Taoran Tang et al. also proposed a well-designed deep learning model to extract the mapping between sound and motion features. Furthermore, they use temporal indexing and masking methods for better performance. As can be seen from previous literature, in recent years, deep neural networks have been the dominant method in the field of computer choreography [10–15].


## 3. Related Theories and Technical Methods

### 3.1. OpenPose Pose Detection Framework

Human posture is a description of a person’s posture in the form of coordinates. Essentially, a human pose is a set of coordinate points that can be connected to describe a person’s pose. OpenPose is one of the most popular multiperson human pose estimation algorithms. Like many bottom-up methods, OpenPose first detects the keypoint coordinates of all people in the image and then assigns the detected keypoints to each corresponding person. In practice, the OpenPose network first extracts features from images using the first few network layers of VGG-19. Next, these features are passed to two parallel branches of convolutional layers. The first branch is used to predict 18 confidence maps, each representing a joint in the human skeleton. The second branch predicts a set containing 38 joint affine fields (Part Affinity Fields, PAFs) that describe the degree of connectivity between joints. A series of steps are then used to optimize the predicted value of each branch. Using the joint confidence map, a bipartite graph can be formed between each joint pair. Then use the PAF value to remove weaker connections in the bipartite graph. Through the above steps, the key points of the human body posture of all the people in the picture can be detected [16].


### 3.2. Image Generation

Many researchers utilize adversarial learning for image-to-image transfer, which aims to transform an input image into another domain space as a synthetic image. Compared toL1 loss, which usually results in blurry images, adversarial loss has become the mainstream choice for many image generation tasks. The reason is that the loss function that the discriminator can learn is trainable and can automatically adapt to the difference between the generated and real images in the target domain. For example, the recent Pix2Pix framework using image-conditioned GANs has been applied to applications in different scenarios, such as converting Google Maps to satellite views or generating synthetic images of cats from user-drawn sketches. The Pix2PixHD framework is based on the Pix2Pix infrastructure and proposes a new method to synthesize high-resolution, realistic images using conditional generative adversarial networks using semantic label maps. Conditional generative adversarial networks have been used in a variety of applications but are limited by the fact that their generated results are usually limited to low resolutions and do not look realistic, while in the Pix2PixHD model, the model is able to generate 2048 × 1024 high-resolution pictures and visually appealing results. The model features a novel adversarial loss, as well as a new multiscale generator and discriminator architecture [17].The Pix2PixHD framework shows us that, with a new and powerful adversarial learning objective function and a new multiscale generator and discriminator architecture as shown in Figure1, it is possible to synthesize 2048 × 1024 resolution and realistic images, which are much smaller than presynthesized images. Images are more visually appealing. The method only needs to achieve the same result with precise training without relying on any human-specified loss or perceptual loss of a pretrained network (e.g., VGGNet).Figure 1
Network structure diagram in Pix2PixHD.


### 3.3. Recurrent Neural Network

The inertial data of the action is a time series structure with sequential order, and changing the input order of the data values of the same size will cause the model results to change. The one-dimensional deformed convolutional neural network is used to realize the processing algorithm of serialized structural data, but the CNN model itself cannot provide temporal correlation, so this paper introduces a cyclic neural network (recurrent neural networks, RNN) and Convolutional Neural Networks [18].The characteristic of the recurrent neural network is that it contains a ring structure as shown in Figure2, which makes the state information circulate in its own network, thus providing a time dependency. RNN networks can handle general sequence problems well, such as a small continuous speech, a sentence, and a text description. However, for relatively long sequences, such as a speech recording, an English song, or an article, the RNN network cannot handle it well. Or sometimes we only need to view the most recent information to perform the current task and do not want to remember all the previous data.Figure 2
Schematic diagram of the ring structure of RNN.The essence of the recurrent neural networkx⟶ encodes the input sequence as a fixed-length hidden state sequence h⟶, where xt⟶=x1,…xt and input time series ht⟶=h1,…ht . The specific meaning refers to the state sequence that changes with time. The update strategy for the hidden state is as follows:(1)ht=fxt，h⟶t−1.f (x, h) for calculating the hidden state is a nonlinear equation, usually a logistic function or a tanh function. From the formula, it can be found that the hidden state information h t at the current moment is not only related to the current input xt, but also related to the state sequence retained before h⟶t−1.Expand the network structure shown in Figure3 on the time axis, and a structure diagram with a clearer dependency relationship can be obtained as shown in Figure 3. Compared with the common neural network, the calculation results are independent of each other. The results are related to the current input and the last calculation result. In this way, the RNN has the characteristics of memorizing the previous calculation results.Figure 3
Schematic diagram of the time-dependent structure of RNN.


## 3.1. OpenPose Pose Detection Framework

Human posture is a description of a person’s posture in the form of coordinates. Essentially, a human pose is a set of coordinate points that can be connected to describe a person’s pose. OpenPose is one of the most popular multiperson human pose estimation algorithms. Like many bottom-up methods, OpenPose first detects the keypoint coordinates of all people in the image and then assigns the detected keypoints to each corresponding person. In practice, the OpenPose network first extracts features from images using the first few network layers of VGG-19. Next, these features are passed to two parallel branches of convolutional layers. The first branch is used to predict 18 confidence maps, each representing a joint in the human skeleton. The second branch predicts a set containing 38 joint affine fields (Part Affinity Fields, PAFs) that describe the degree of connectivity between joints. A series of steps are then used to optimize the predicted value of each branch. Using the joint confidence map, a bipartite graph can be formed between each joint pair. Then use the PAF value to remove weaker connections in the bipartite graph. Through the above steps, the key points of the human body posture of all the people in the picture can be detected [16].


## 3.2. Image Generation

Many researchers utilize adversarial learning for image-to-image transfer, which aims to transform an input image into another domain space as a synthetic image. Compared toL1 loss, which usually results in blurry images, adversarial loss has become the mainstream choice for many image generation tasks. The reason is that the loss function that the discriminator can learn is trainable and can automatically adapt to the difference between the generated and real images in the target domain. For example, the recent Pix2Pix framework using image-conditioned GANs has been applied to applications in different scenarios, such as converting Google Maps to satellite views or generating synthetic images of cats from user-drawn sketches. The Pix2PixHD framework is based on the Pix2Pix infrastructure and proposes a new method to synthesize high-resolution, realistic images using conditional generative adversarial networks using semantic label maps. Conditional generative adversarial networks have been used in a variety of applications but are limited by the fact that their generated results are usually limited to low resolutions and do not look realistic, while in the Pix2PixHD model, the model is able to generate 2048 × 1024 high-resolution pictures and visually appealing results. The model features a novel adversarial loss, as well as a new multiscale generator and discriminator architecture [17].The Pix2PixHD framework shows us that, with a new and powerful adversarial learning objective function and a new multiscale generator and discriminator architecture as shown in Figure1, it is possible to synthesize 2048 × 1024 resolution and realistic images, which are much smaller than presynthesized images. Images are more visually appealing. The method only needs to achieve the same result with precise training without relying on any human-specified loss or perceptual loss of a pretrained network (e.g., VGGNet).Figure 1
Network structure diagram in Pix2PixHD.


## 3.3. Recurrent Neural Network

The inertial data of the action is a time series structure with sequential order, and changing the input order of the data values of the same size will cause the model results to change. The one-dimensional deformed convolutional neural network is used to realize the processing algorithm of serialized structural data, but the CNN model itself cannot provide temporal correlation, so this paper introduces a cyclic neural network (recurrent neural networks, RNN) and Convolutional Neural Networks [18].The characteristic of the recurrent neural network is that it contains a ring structure as shown in Figure2, which makes the state information circulate in its own network, thus providing a time dependency. RNN networks can handle general sequence problems well, such as a small continuous speech, a sentence, and a text description. However, for relatively long sequences, such as a speech recording, an English song, or an article, the RNN network cannot handle it well. Or sometimes we only need to view the most recent information to perform the current task and do not want to remember all the previous data.Figure 2
Schematic diagram of the ring structure of RNN.The essence of the recurrent neural networkx⟶ encodes the input sequence as a fixed-length hidden state sequence h⟶, where xt⟶=x1,…xt and input time series ht⟶=h1,…ht . The specific meaning refers to the state sequence that changes with time. The update strategy for the hidden state is as follows:(1)ht=fxt，h⟶t−1.f (x, h) for calculating the hidden state is a nonlinear equation, usually a logistic function or a tanh function. From the formula, it can be found that the hidden state information h t at the current moment is not only related to the current input xt, but also related to the state sequence retained before h⟶t−1.Expand the network structure shown in Figure3 on the time axis, and a structure diagram with a clearer dependency relationship can be obtained as shown in Figure 3. Compared with the common neural network, the calculation results are independent of each other. The results are related to the current input and the last calculation result. In this way, the RNN has the characteristics of memorizing the previous calculation results.Figure 3
Schematic diagram of the time-dependent structure of RNN.


## 4. Design of Dance Movement Generation System Based on Recurrent Neural Network

### 4.1. System Structure Design

The key to realizing the dance generation application based on generative adversarial network lies in the patch-based pixel generation in the generative network and the multiscale image discrimination in the discriminant network and the final loss function design. The overall structure of this application is shown in Figure4 [19].(1)
Interface layer: The interface layer is mainly used for the interaction between the user and the system, and all functions need to be as simple as possible. This application interface mainly includes various controls and view modules, such as selecting video buttons, displaying video views, generating dance buttons, and text prompt views. The application interface will directly display the original dance video and the generated dance video to the user, compare the synthesis effect, and generate corresponding indicator descriptions. Simple and elegant interface design will bring users a good user experience.(2)
Logic layer: The logic layer is often the core of the entire application system, including the realization of core functions and the connection between the upper and lower layers. The main functions include reading local video or collecting video data from the current camera. Collecting data is an indispensable function for generating models; in addition, the system needs to call other open source projects to extract skeleton information from the data. More importantly, inputting the skeleton diagram into the trained generation model to get the dance generated by the target is the core of the system application; finally, the system will show the generated dance video to the user, letting users experience the thrill of generating dance videos without having to learn to dance by themselves.(3)
Data layer: The data layer is mainly used to store and read video data and also includes model storage and generation of video storage and reading. Since this system is an offline system, the model is too large and the training time is long, so online training and generation cannot be realized at present. Therefore, the system must store the pretrained character action model and then generate the dance video of the corresponding character according to the input to the skeleton model. The generated dance video will be stored in the specified directory.Figure 4
Overall system structure.


### 4.2. System Algorithm Design

The task of this application is to simultaneously generate actions between different characters, i.e., transition from a given pose to a desired pose while maintaining important appearance details of the target character. In this application, the system will apply the pose skeleton map (motion representation) to different target subjects to generate new motions. As an intermediate layer, the skeleton map conveys the movement information of the character, and the skeleton map is not restricted by the figure and clothing of the character, so in principle, this application can perform arbitrary actions on any object [9].


#### 4.2.1. Generator Design

In order to improve the user control ability of the generative adversarial network and still improve the resolution and quality of the generated pictures of the generative adversarial network, we first use paired data for training. As shown in the previous section, the real sample pair is (x, y). The fake sample pair is (G(x), y). This improvement provides the generation network with the correspondence between the appearance of the character and the skeleton, and this kind of application has a wide range of applications. Similar applications include generating pictures from hand-drawn sketches, coloring grayscale images, and automatic coloring. In order to solve the problem of the quality and resolution of the generated image, we use the Pix2PixHD structure to generate a high-resolution image, as shown in Figure 5. The most important improvement is to use a pyramid method, first output low-resolution pictures, then use the previously output low-resolution pictures as the input of another generation network, and continue to generate better detailed and higher-resolution pictures. What is used in Pix2PixHD is a coarse-to-fine generation network, but this application does not use this structure but only borrows the “coarse” network and uses its fine network. The generator network mainly generates 512∗1024 pictures, including a series of self-encoding modules with residuals.Figure 5
Generator network.


#### 4.2.2. Discriminator Design

In order to be able to distinguish high-resolution real and synthetic images, the discriminator network needs a large receptive field. A deeper network or a larger size convolution kernel can have a larger receptive field, but this is also likely to lead to overfitting of the network and more GPU memory required for too many network parameters. In order to solve this problem in this system, we use the multiscale discriminator proposed in the Pix2PixHD paper. We use 3 discriminators, each of which has the same network structure but operates on different image scales {d1, d2, d3}. Specifically, we downsample real and synthetic high-resolution images by a factor of 2 and 4 to generate pyramid image data at 3 scales. The discriminators d1, d2, and d3 are trained to credit real and synthetic images at three different scales. The discriminator has the same structure, but an operation at the coarsest scale has the largest receptive field. It has a global view of the image and guides the generator to generate globally continuous images. On the other hand, the discriminator at the smallest scale encourages the generator to generate finer details. This also makes it simpler to train a coarse-to-fine generator, and scaling a lower-resolution model to meet higher-resolution only requires adding a discriminator at the finest-grained layer, rather than retraining from the skeleton. The input of each discriminative network is images of different scales, and finally the results are fed into a patch-based generative adversarial loss. In this way, the purpose of the discriminator network having a larger receptive field is achieved. The original generative adversarial network has also evolved into a multitask generative adversarial network (see loss function design for details). The multiscale discriminator only has different input image sizes, and other structures are the same, as shown in Figure 6 [20–22].Figure 6
Discriminator network.


### 4.3. System Structure Design

We collect dance source videos as well as target character action videos in different ways. The data structure design of this system is shown in Figure7, which is divided into training data and test data. In order to learn the appearance of the target character in various poses, we need to capture a sufficient range of actions in the target video, including the character turning around, raising his legs back and forth, hand movements, bending over and turning his head, etc. And in order to ensure the quality of the video, we need to shoot a video of the target person for about 20 minutes, and the video frame rate is 120 fps.Figure 7
Dance data structure diagram.However, the requirements of the dance source video are not as stringent as that of the training video. We only need to meet the requirements that the source video cannot be shaken during recording and that the distance between the screen and the character is kept reasonable, and the gesture detector can detect the movement information of the character. Training video screenshots and redrawn skeletons are shown in Figure8.Figure 8
Screenshot of training video and redrawn skeleton diagram.The key points of the character body mainly include 21 key points of limbs, such as nose, ears, neck, shoulders, arms, hips, knees, and other joints. The key points of the character's hand include 20 key points in each of the left and right hands, mainly including the joints of the five fingers. The key points of the face mainly include 5 key points, mainly the two eyeballs, the nose, the middle, and the chin.


## 4.1. System Structure Design

The key to realizing the dance generation application based on generative adversarial network lies in the patch-based pixel generation in the generative network and the multiscale image discrimination in the discriminant network and the final loss function design. The overall structure of this application is shown in Figure4 [19].(1)
Interface layer: The interface layer is mainly used for the interaction between the user and the system, and all functions need to be as simple as possible. This application interface mainly includes various controls and view modules, such as selecting video buttons, displaying video views, generating dance buttons, and text prompt views. The application interface will directly display the original dance video and the generated dance video to the user, compare the synthesis effect, and generate corresponding indicator descriptions. Simple and elegant interface design will bring users a good user experience.(2)
Logic layer: The logic layer is often the core of the entire application system, including the realization of core functions and the connection between the upper and lower layers. The main functions include reading local video or collecting video data from the current camera. Collecting data is an indispensable function for generating models; in addition, the system needs to call other open source projects to extract skeleton information from the data. More importantly, inputting the skeleton diagram into the trained generation model to get the dance generated by the target is the core of the system application; finally, the system will show the generated dance video to the user, letting users experience the thrill of generating dance videos without having to learn to dance by themselves.(3)
Data layer: The data layer is mainly used to store and read video data and also includes model storage and generation of video storage and reading. Since this system is an offline system, the model is too large and the training time is long, so online training and generation cannot be realized at present. Therefore, the system must store the pretrained character action model and then generate the dance video of the corresponding character according to the input to the skeleton model. The generated dance video will be stored in the specified directory.Figure 4
Overall system structure.


## 4.2. System Algorithm Design

The task of this application is to simultaneously generate actions between different characters, i.e., transition from a given pose to a desired pose while maintaining important appearance details of the target character. In this application, the system will apply the pose skeleton map (motion representation) to different target subjects to generate new motions. As an intermediate layer, the skeleton map conveys the movement information of the character, and the skeleton map is not restricted by the figure and clothing of the character, so in principle, this application can perform arbitrary actions on any object [9].


### 4.2.1. Generator Design

In order to improve the user control ability of the generative adversarial network and still improve the resolution and quality of the generated pictures of the generative adversarial network, we first use paired data for training. As shown in the previous section, the real sample pair is (x, y). The fake sample pair is (G(x), y). This improvement provides the generation network with the correspondence between the appearance of the character and the skeleton, and this kind of application has a wide range of applications. Similar applications include generating pictures from hand-drawn sketches, coloring grayscale images, and automatic coloring. In order to solve the problem of the quality and resolution of the generated image, we use the Pix2PixHD structure to generate a high-resolution image, as shown in Figure 5. The most important improvement is to use a pyramid method, first output low-resolution pictures, then use the previously output low-resolution pictures as the input of another generation network, and continue to generate better detailed and higher-resolution pictures. What is used in Pix2PixHD is a coarse-to-fine generation network, but this application does not use this structure but only borrows the “coarse” network and uses its fine network. The generator network mainly generates 512∗1024 pictures, including a series of self-encoding modules with residuals.Figure 5
Generator network.


### 4.2.2. Discriminator Design

In order to be able to distinguish high-resolution real and synthetic images, the discriminator network needs a large receptive field. A deeper network or a larger size convolution kernel can have a larger receptive field, but this is also likely to lead to overfitting of the network and more GPU memory required for too many network parameters. In order to solve this problem in this system, we use the multiscale discriminator proposed in the Pix2PixHD paper. We use 3 discriminators, each of which has the same network structure but operates on different image scales {d1, d2, d3}. Specifically, we downsample real and synthetic high-resolution images by a factor of 2 and 4 to generate pyramid image data at 3 scales. The discriminators d1, d2, and d3 are trained to credit real and synthetic images at three different scales. The discriminator has the same structure, but an operation at the coarsest scale has the largest receptive field. It has a global view of the image and guides the generator to generate globally continuous images. On the other hand, the discriminator at the smallest scale encourages the generator to generate finer details. This also makes it simpler to train a coarse-to-fine generator, and scaling a lower-resolution model to meet higher-resolution only requires adding a discriminator at the finest-grained layer, rather than retraining from the skeleton. The input of each discriminative network is images of different scales, and finally the results are fed into a patch-based generative adversarial loss. In this way, the purpose of the discriminator network having a larger receptive field is achieved. The original generative adversarial network has also evolved into a multitask generative adversarial network (see loss function design for details). The multiscale discriminator only has different input image sizes, and other structures are the same, as shown in Figure 6 [20–22].Figure 6
Discriminator network.


## 4.2.1. Generator Design

In order to improve the user control ability of the generative adversarial network and still improve the resolution and quality of the generated pictures of the generative adversarial network, we first use paired data for training. As shown in the previous section, the real sample pair is (x, y). The fake sample pair is (G(x), y). This improvement provides the generation network with the correspondence between the appearance of the character and the skeleton, and this kind of application has a wide range of applications. Similar applications include generating pictures from hand-drawn sketches, coloring grayscale images, and automatic coloring. In order to solve the problem of the quality and resolution of the generated image, we use the Pix2PixHD structure to generate a high-resolution image, as shown in Figure 5. The most important improvement is to use a pyramid method, first output low-resolution pictures, then use the previously output low-resolution pictures as the input of another generation network, and continue to generate better detailed and higher-resolution pictures. What is used in Pix2PixHD is a coarse-to-fine generation network, but this application does not use this structure but only borrows the “coarse” network and uses its fine network. The generator network mainly generates 512∗1024 pictures, including a series of self-encoding modules with residuals.Figure 5
Generator network.


## 4.2.2. Discriminator Design

In order to be able to distinguish high-resolution real and synthetic images, the discriminator network needs a large receptive field. A deeper network or a larger size convolution kernel can have a larger receptive field, but this is also likely to lead to overfitting of the network and more GPU memory required for too many network parameters. In order to solve this problem in this system, we use the multiscale discriminator proposed in the Pix2PixHD paper. We use 3 discriminators, each of which has the same network structure but operates on different image scales {d1, d2, d3}. Specifically, we downsample real and synthetic high-resolution images by a factor of 2 and 4 to generate pyramid image data at 3 scales. The discriminators d1, d2, and d3 are trained to credit real and synthetic images at three different scales. The discriminator has the same structure, but an operation at the coarsest scale has the largest receptive field. It has a global view of the image and guides the generator to generate globally continuous images. On the other hand, the discriminator at the smallest scale encourages the generator to generate finer details. This also makes it simpler to train a coarse-to-fine generator, and scaling a lower-resolution model to meet higher-resolution only requires adding a discriminator at the finest-grained layer, rather than retraining from the skeleton. The input of each discriminative network is images of different scales, and finally the results are fed into a patch-based generative adversarial loss. In this way, the purpose of the discriminator network having a larger receptive field is achieved. The original generative adversarial network has also evolved into a multitask generative adversarial network (see loss function design for details). The multiscale discriminator only has different input image sizes, and other structures are the same, as shown in Figure 6 [20–22].Figure 6
Discriminator network.


## 4.3. System Structure Design

We collect dance source videos as well as target character action videos in different ways. The data structure design of this system is shown in Figure7, which is divided into training data and test data. In order to learn the appearance of the target character in various poses, we need to capture a sufficient range of actions in the target video, including the character turning around, raising his legs back and forth, hand movements, bending over and turning his head, etc. And in order to ensure the quality of the video, we need to shoot a video of the target person for about 20 minutes, and the video frame rate is 120 fps.Figure 7
Dance data structure diagram.However, the requirements of the dance source video are not as stringent as that of the training video. We only need to meet the requirements that the source video cannot be shaken during recording and that the distance between the screen and the character is kept reasonable, and the gesture detector can detect the movement information of the character. Training video screenshots and redrawn skeletons are shown in Figure8.Figure 8
Screenshot of training video and redrawn skeleton diagram.The key points of the character body mainly include 21 key points of limbs, such as nose, ears, neck, shoulders, arms, hips, knees, and other joints. The key points of the character's hand include 20 key points in each of the left and right hands, mainly including the joints of the five fingers. The key points of the face mainly include 5 key points, mainly the two eyeballs, the nose, the middle, and the chin.


## 5. Implementation and Testing of Dance Movement Generation System Based on Recurrent Neural Network

### 5.1. System Development Environment

The training and generation of this application are run on the server, and the configuration of the runtime development environment is shown in Table1. For the core algorithm, a large amount of video memory is required. This time, 8 tesla p100 GPUs are used for training, and the training time is 5∗24 hours. Since the system model needs to be trained for a long time, the displayed samples are all models trained in advance.Table 1
System development and runtime environment.
Host parametersOS: Centos CPU: Intel E7 system memory: 128 GB GPU: Tesla p100∗8 GPU memory: 16 GBCore algorithm development environmentPyTorch version: 0.4 CUDA version: 8.0 IDE: vim development language: Python3Application test development environmentCPU: Intel E7 PyTorch version: 0.4 GPU: Tesla p100 GPU memory: 16 GB development language: Python3 GUI: PyQt5


### 5.2. Function Realization of Dance Movement Generation System

According to the previous requirement analysis and structural design, this chapter specifically implements the dance generation application based on the recurrent neural network. The core of this article is that the danceTrans algorithm is developed and implemented based on the PyTorch deep learning framework and the open source pose detection OpenPose and image processing open source library OpenCV. According to the system function, it is divided into three parts: data input and output, skeleton extraction, and dance generation.


#### 5.2.1. Implementation of Dance Generation Algorithm

The core algorithm danceTrans of this application mainly designs multiple loss functions to jointly control the output and uses some training techniques to train the model. First, the loss function in the danceTrans algorithm is mainly divided into generator loss and discriminator loss. The generator loss mainly includes adversarial loss, VGG-based perceptual loss, and feature matching loss. Adversarial losses have become a popular choice for many image-to-image tasks because the discriminator can learn a trainable loss function and automatically adapt to the differences between generated and real images in the target domain. The feature matching loss is more important to ensure that the generated model does not oscillate too much. And through specific experiments, it is proved that increasing the VGG perceptual loss can slightly improve the generation results in some cases. The loss of the discriminator mainly includes the loss of real image pairs and the loss of fake image pairs, that is, the loss caused by discriminating real images as fake and discriminating fake images into real image pairs.In Figure9, the loss value saved during the training process is visualized. The content of the figure is the visualized image after the superposition of all generator losses. It can be seen that the generator loss will shake greatly during the training of the generative adversarial network. After several rounds of training, there will be significant relief. However, there is often a sudden jump in a loss value during the training process. We specifically analyze the problem here, mainly because when synthesizing the back image of the character, the distribution of the front image is quite different, so the front and back of the good character are not distinguished when synthesizing, resulting in jitter in the loss value there.Figure 9
Generator loss curves.From the discriminator loss graph, which is shown in Figure10, it can be seen that the discriminator loss has a large loss during the initial training, but after 15 rounds of iterative evolution, the loss value stabilizes within an interval.Figure 10
Discriminator loss curve.


#### 5.2.2. Implementation of Dance Data Input and Output Module

The video input of this application can be done in two ways; one is to read the local video, the other is to call the system camera to record the video and store the video locally. Therefore, it is necessary to monitor the data entry button. When the user clicks the record video button, the system will detect the camera device that can be called by the system and activate the device for video recording. When the user's video recording is completed, the user needs to select the video storage address and specify the video name. If the new video file fails, an error message will be returned. When the user chooses to directly read the local video data, the user needs to specify the video file and return the video URL. This application reads the video file according to the URL specified by the user. If the selected video cannot be read or the format of the read video is wrong, an error message will be returned. The video output method of this application is to synthesize the generated image collection into a video file and store it locally. When the dance generation program is completed, the system will generate a new composite image set. The user needs to specify the stored video file and the video file name, and the system will create a new video file and store it. If the new file fails, the system will return an error message.


#### 5.2.3. Implementation of Dance Action Skeleton Detection Module

After the video file collection is completed, the application first uses the FFmpeg tool to cut the video file into corresponding frame files and save them as the gt folder; then check the gt folder, including the number of files, file number continuity, and image size; then input all the image files in the gt folder into the OpenPose program component and specify the output path to get the character skeleton JSON file, including 70 face key points, 30 limb key points, and 21 hand key points; then, the application generates the corresponding self-drawn skeleton map set according to the generated JSON folder; for the test, after we get the character skeleton JSON, we use the mean value compensation in the time dimension for the coordinates of each key point (x, y), full and smooth, and then generate the corresponding skeleton map file.


#### 5.2.4. Implementation of Dance Generation Module

We now detail our complete training pipeline as shown in the training setup in Figure11. Given a frame y from the original target video, we use a pose detector P to obtain the corresponding pose skeleton map x = P(y). During training, we use the corresponding (x, y) pairs to learn a generator G that maps a given pose skeleton map x to a target person image. By adversarial training with a pretrained VGGNet with a discriminator D and a visual perceptual reconstruction loss, we optimize the generated output G (x) to resemble the real image target subject frame y. The discriminator D tries to distinguish between “real” image pairs (i.e., pose skeleton map x, real person image y) and “fake” image pairs (i.e., pose skeleton map x, generator model output G (x)). In the training process, we first spliced the real image of the character and the redrawn skeleton image and spliced according to the color channel of the image. Then the number of channels of the real image of the character is RGB channels, and the skeleton image is also an RGB image, so for the final model of our model the input data is a 6-dimensional 512∗1024 combined image. We expect the final model to output a RGB three-channel reconstructed person image. The input of the discriminator is only the RGB real image and the synthesized image of the person. Therefore, the input data format of the discriminator and the generator will be different, and the discriminator finally outputs only 0 or 1 values to represent real and fake images. Our generative setup is shown in the structural setup in Figure 11. Similar to training, the pose detector P starts from the source frame y'. Generate the extracted pose information, and then input it to the generator to get the final dance generated result image.Figure 11
Model training flowchart.To better align the source pose with the target’s shooting setup, we generate the source skeleton map to a specified location to make it more consistent with the pose in the target videox. We then pass the pose stick figure x into our trained model G to obtain an image G (x) of our target person corresponding to the original image of the source y', as shown in Figure 12.Figure 12
Model generation structure diagram.


#### 5.2.5. Dance Generation Application Test

According to the requirement analysis and structural design of the dance generation application, this section introduces the testing of each functional module in the testing phase in detail. For each test case, we need to repeat the test for each function multiple times and require all tests to pass to determine that the test case is passed.Next, we demonstrate how this application performs the effect of dance generation, as shown in Figures13 and 14; when the user clicks the open button and selects the dance video, the system will cut the video into frame-by-frame images and detect the skeleton information of the characters. After getting the skeleton information, the system will automatically redraw the skeleton of the character and see the redrawn skeleton image passed into the trained model to display the final composite image. It can be seen from this image that when the character generation effect is not good, most of the errors occur in the skeleton extraction, including false detection and missed detection. When the skeleton is relatively clear, the character generation effect is very good. And we do temporal smoothing when compositing the video, which makes our composite video look very coherent.Figure 13
Synthesis results of 50 iterations.Figure 14
The synthetic result of 100 iterations.Finally, the training results of the multiperson model are drawn, and Figure15 shows the process of generating the same actions for four people from the actions shown in the first row. The first row is the 6 frames of images captured from the source video, the second row is the redrawn character skeleton diagram extracted from the captured images, and the third to sixth rows are the set of action images generated by the target character. The results show that our model can also train well in multiperson (multisample) situations. As can be seen from the figure, our model can synthesize better effects for different character skeletons and has nothing to do with the character’s body and background.Figure 15
Dance generation results.


## 5.1. System Development Environment

The training and generation of this application are run on the server, and the configuration of the runtime development environment is shown in Table1. For the core algorithm, a large amount of video memory is required. This time, 8 tesla p100 GPUs are used for training, and the training time is 5∗24 hours. Since the system model needs to be trained for a long time, the displayed samples are all models trained in advance.Table 1
System development and runtime environment.
Host parametersOS: Centos CPU: Intel E7 system memory: 128 GB GPU: Tesla p100∗8 GPU memory: 16 GBCore algorithm development environmentPyTorch version: 0.4 CUDA version: 8.0 IDE: vim development language: Python3Application test development environmentCPU: Intel E7 PyTorch version: 0.4 GPU: Tesla p100 GPU memory: 16 GB development language: Python3 GUI: PyQt5


## 5.2. Function Realization of Dance Movement Generation System

According to the previous requirement analysis and structural design, this chapter specifically implements the dance generation application based on the recurrent neural network. The core of this article is that the danceTrans algorithm is developed and implemented based on the PyTorch deep learning framework and the open source pose detection OpenPose and image processing open source library OpenCV. According to the system function, it is divided into three parts: data input and output, skeleton extraction, and dance generation.


### 5.2.1. Implementation of Dance Generation Algorithm

The core algorithm danceTrans of this application mainly designs multiple loss functions to jointly control the output and uses some training techniques to train the model. First, the loss function in the danceTrans algorithm is mainly divided into generator loss and discriminator loss. The generator loss mainly includes adversarial loss, VGG-based perceptual loss, and feature matching loss. Adversarial losses have become a popular choice for many image-to-image tasks because the discriminator can learn a trainable loss function and automatically adapt to the differences between generated and real images in the target domain. The feature matching loss is more important to ensure that the generated model does not oscillate too much. And through specific experiments, it is proved that increasing the VGG perceptual loss can slightly improve the generation results in some cases. The loss of the discriminator mainly includes the loss of real image pairs and the loss of fake image pairs, that is, the loss caused by discriminating real images as fake and discriminating fake images into real image pairs.In Figure9, the loss value saved during the training process is visualized. The content of the figure is the visualized image after the superposition of all generator losses. It can be seen that the generator loss will shake greatly during the training of the generative adversarial network. After several rounds of training, there will be significant relief. However, there is often a sudden jump in a loss value during the training process. We specifically analyze the problem here, mainly because when synthesizing the back image of the character, the distribution of the front image is quite different, so the front and back of the good character are not distinguished when synthesizing, resulting in jitter in the loss value there.Figure 9
Generator loss curves.From the discriminator loss graph, which is shown in Figure10, it can be seen that the discriminator loss has a large loss during the initial training, but after 15 rounds of iterative evolution, the loss value stabilizes within an interval.Figure 10
Discriminator loss curve.


### 5.2.2. Implementation of Dance Data Input and Output Module

The video input of this application can be done in two ways; one is to read the local video, the other is to call the system camera to record the video and store the video locally. Therefore, it is necessary to monitor the data entry button. When the user clicks the record video button, the system will detect the camera device that can be called by the system and activate the device for video recording. When the user's video recording is completed, the user needs to select the video storage address and specify the video name. If the new video file fails, an error message will be returned. When the user chooses to directly read the local video data, the user needs to specify the video file and return the video URL. This application reads the video file according to the URL specified by the user. If the selected video cannot be read or the format of the read video is wrong, an error message will be returned. The video output method of this application is to synthesize the generated image collection into a video file and store it locally. When the dance generation program is completed, the system will generate a new composite image set. The user needs to specify the stored video file and the video file name, and the system will create a new video file and store it. If the new file fails, the system will return an error message.


### 5.2.3. Implementation of Dance Action Skeleton Detection Module

After the video file collection is completed, the application first uses the FFmpeg tool to cut the video file into corresponding frame files and save them as the gt folder; then check the gt folder, including the number of files, file number continuity, and image size; then input all the image files in the gt folder into the OpenPose program component and specify the output path to get the character skeleton JSON file, including 70 face key points, 30 limb key points, and 21 hand key points; then, the application generates the corresponding self-drawn skeleton map set according to the generated JSON folder; for the test, after we get the character skeleton JSON, we use the mean value compensation in the time dimension for the coordinates of each key point (x, y), full and smooth, and then generate the corresponding skeleton map file.


### 5.2.4. Implementation of Dance Generation Module

We now detail our complete training pipeline as shown in the training setup in Figure11. Given a frame y from the original target video, we use a pose detector P to obtain the corresponding pose skeleton map x = P(y). During training, we use the corresponding (x, y) pairs to learn a generator G that maps a given pose skeleton map x to a target person image. By adversarial training with a pretrained VGGNet with a discriminator D and a visual perceptual reconstruction loss, we optimize the generated output G (x) to resemble the real image target subject frame y. The discriminator D tries to distinguish between “real” image pairs (i.e., pose skeleton map x, real person image y) and “fake” image pairs (i.e., pose skeleton map x, generator model output G (x)). In the training process, we first spliced the real image of the character and the redrawn skeleton image and spliced according to the color channel of the image. Then the number of channels of the real image of the character is RGB channels, and the skeleton image is also an RGB image, so for the final model of our model the input data is a 6-dimensional 512∗1024 combined image. We expect the final model to output a RGB three-channel reconstructed person image. The input of the discriminator is only the RGB real image and the synthesized image of the person. Therefore, the input data format of the discriminator and the generator will be different, and the discriminator finally outputs only 0 or 1 values to represent real and fake images. Our generative setup is shown in the structural setup in Figure 11. Similar to training, the pose detector P starts from the source frame y'. Generate the extracted pose information, and then input it to the generator to get the final dance generated result image.Figure 11
Model training flowchart.To better align the source pose with the target’s shooting setup, we generate the source skeleton map to a specified location to make it more consistent with the pose in the target videox. We then pass the pose stick figure x into our trained model G to obtain an image G (x) of our target person corresponding to the original image of the source y', as shown in Figure 12.Figure 12
Model generation structure diagram.


### 5.2.5. Dance Generation Application Test

According to the requirement analysis and structural design of the dance generation application, this section introduces the testing of each functional module in the testing phase in detail. For each test case, we need to repeat the test for each function multiple times and require all tests to pass to determine that the test case is passed.Next, we demonstrate how this application performs the effect of dance generation, as shown in Figures13 and 14; when the user clicks the open button and selects the dance video, the system will cut the video into frame-by-frame images and detect the skeleton information of the characters. After getting the skeleton information, the system will automatically redraw the skeleton of the character and see the redrawn skeleton image passed into the trained model to display the final composite image. It can be seen from this image that when the character generation effect is not good, most of the errors occur in the skeleton extraction, including false detection and missed detection. When the skeleton is relatively clear, the character generation effect is very good. And we do temporal smoothing when compositing the video, which makes our composite video look very coherent.Figure 13
Synthesis results of 50 iterations.Figure 14
The synthetic result of 100 iterations.Finally, the training results of the multiperson model are drawn, and Figure15 shows the process of generating the same actions for four people from the actions shown in the first row. The first row is the 6 frames of images captured from the source video, the second row is the redrawn character skeleton diagram extracted from the captured images, and the third to sixth rows are the set of action images generated by the target character. The results show that our model can also train well in multiperson (multisample) situations. As can be seen from the figure, our model can synthesize better effects for different character skeletons and has nothing to do with the character’s body and background.Figure 15
Dance generation results.


## 5.2.1. Implementation of Dance Generation Algorithm

The core algorithm danceTrans of this application mainly designs multiple loss functions to jointly control the output and uses some training techniques to train the model. First, the loss function in the danceTrans algorithm is mainly divided into generator loss and discriminator loss. The generator loss mainly includes adversarial loss, VGG-based perceptual loss, and feature matching loss. Adversarial losses have become a popular choice for many image-to-image tasks because the discriminator can learn a trainable loss function and automatically adapt to the differences between generated and real images in the target domain. The feature matching loss is more important to ensure that the generated model does not oscillate too much. And through specific experiments, it is proved that increasing the VGG perceptual loss can slightly improve the generation results in some cases. The loss of the discriminator mainly includes the loss of real image pairs and the loss of fake image pairs, that is, the loss caused by discriminating real images as fake and discriminating fake images into real image pairs.In Figure9, the loss value saved during the training process is visualized. The content of the figure is the visualized image after the superposition of all generator losses. It can be seen that the generator loss will shake greatly during the training of the generative adversarial network. After several rounds of training, there will be significant relief. However, there is often a sudden jump in a loss value during the training process. We specifically analyze the problem here, mainly because when synthesizing the back image of the character, the distribution of the front image is quite different, so the front and back of the good character are not distinguished when synthesizing, resulting in jitter in the loss value there.Figure 9
Generator loss curves.From the discriminator loss graph, which is shown in Figure10, it can be seen that the discriminator loss has a large loss during the initial training, but after 15 rounds of iterative evolution, the loss value stabilizes within an interval.Figure 10
Discriminator loss curve.


## 5.2.2. Implementation of Dance Data Input and Output Module

The video input of this application can be done in two ways; one is to read the local video, the other is to call the system camera to record the video and store the video locally. Therefore, it is necessary to monitor the data entry button. When the user clicks the record video button, the system will detect the camera device that can be called by the system and activate the device for video recording. When the user's video recording is completed, the user needs to select the video storage address and specify the video name. If the new video file fails, an error message will be returned. When the user chooses to directly read the local video data, the user needs to specify the video file and return the video URL. This application reads the video file according to the URL specified by the user. If the selected video cannot be read or the format of the read video is wrong, an error message will be returned. The video output method of this application is to synthesize the generated image collection into a video file and store it locally. When the dance generation program is completed, the system will generate a new composite image set. The user needs to specify the stored video file and the video file name, and the system will create a new video file and store it. If the new file fails, the system will return an error message.


## 5.2.3. Implementation of Dance Action Skeleton Detection Module

After the video file collection is completed, the application first uses the FFmpeg tool to cut the video file into corresponding frame files and save them as the gt folder; then check the gt folder, including the number of files, file number continuity, and image size; then input all the image files in the gt folder into the OpenPose program component and specify the output path to get the character skeleton JSON file, including 70 face key points, 30 limb key points, and 21 hand key points; then, the application generates the corresponding self-drawn skeleton map set according to the generated JSON folder; for the test, after we get the character skeleton JSON, we use the mean value compensation in the time dimension for the coordinates of each key point (x, y), full and smooth, and then generate the corresponding skeleton map file.


## 5.2.4. Implementation of Dance Generation Module

We now detail our complete training pipeline as shown in the training setup in Figure11. Given a frame y from the original target video, we use a pose detector P to obtain the corresponding pose skeleton map x = P(y). During training, we use the corresponding (x, y) pairs to learn a generator G that maps a given pose skeleton map x to a target person image. By adversarial training with a pretrained VGGNet with a discriminator D and a visual perceptual reconstruction loss, we optimize the generated output G (x) to resemble the real image target subject frame y. The discriminator D tries to distinguish between “real” image pairs (i.e., pose skeleton map x, real person image y) and “fake” image pairs (i.e., pose skeleton map x, generator model output G (x)). In the training process, we first spliced the real image of the character and the redrawn skeleton image and spliced according to the color channel of the image. Then the number of channels of the real image of the character is RGB channels, and the skeleton image is also an RGB image, so for the final model of our model the input data is a 6-dimensional 512∗1024 combined image. We expect the final model to output a RGB three-channel reconstructed person image. The input of the discriminator is only the RGB real image and the synthesized image of the person. Therefore, the input data format of the discriminator and the generator will be different, and the discriminator finally outputs only 0 or 1 values to represent real and fake images. Our generative setup is shown in the structural setup in Figure 11. Similar to training, the pose detector P starts from the source frame y'. Generate the extracted pose information, and then input it to the generator to get the final dance generated result image.Figure 11
Model training flowchart.To better align the source pose with the target’s shooting setup, we generate the source skeleton map to a specified location to make it more consistent with the pose in the target videox. We then pass the pose stick figure x into our trained model G to obtain an image G (x) of our target person corresponding to the original image of the source y', as shown in Figure 12.Figure 12
Model generation structure diagram.


## 5.2.5. Dance Generation Application Test

According to the requirement analysis and structural design of the dance generation application, this section introduces the testing of each functional module in the testing phase in detail. For each test case, we need to repeat the test for each function multiple times and require all tests to pass to determine that the test case is passed.Next, we demonstrate how this application performs the effect of dance generation, as shown in Figures13 and 14; when the user clicks the open button and selects the dance video, the system will cut the video into frame-by-frame images and detect the skeleton information of the characters. After getting the skeleton information, the system will automatically redraw the skeleton of the character and see the redrawn skeleton image passed into the trained model to display the final composite image. It can be seen from this image that when the character generation effect is not good, most of the errors occur in the skeleton extraction, including false detection and missed detection. When the skeleton is relatively clear, the character generation effect is very good. And we do temporal smoothing when compositing the video, which makes our composite video look very coherent.Figure 13
Synthesis results of 50 iterations.Figure 14
The synthetic result of 100 iterations.Finally, the training results of the multiperson model are drawn, and Figure15 shows the process of generating the same actions for four people from the actions shown in the first row. The first row is the 6 frames of images captured from the source video, the second row is the redrawn character skeleton diagram extracted from the captured images, and the third to sixth rows are the set of action images generated by the target character. The results show that our model can also train well in multiperson (multisample) situations. As can be seen from the figure, our model can synthesize better effects for different character skeletons and has nothing to do with the character’s body and background.Figure 15
Dance generation results.


## 6. Conclusion

At present, some existing models for generating dance sequences cannot well represent the strong correlation between music and dance, and there is no relatively complete method for evaluating dance synthesis models. This paper proposes a general video generation framework for human dance generation. With well-designed generator and discriminator networks and various loss functions as objective functions, we can synthesize high-resolution, realistic dance videos. Extensive experiments show that the results generated by our application are significantly better than those generated by other methods.



---

*Source: 8455961-2022-08-04.xml*