# Analysis of Guzheng Music Style Transformation Based on Generative Adversarial Networks

Authors: Lei Zhang
Journal: Mobile Information Systems
 (2022)

---

## Abstract

With the emergence of more and more new music styles, Guzheng, a traditional national musical instrument that has been handed down for thousands of years, also faces new challenges. However, with the development of the times and aesthetics, composers have created various modern Guzheng music, thus injecting new vitality into the development of Guzheng. In order to study the mutual conversion between Guzheng music and other music, this paper proposes a Guzheng music conversion model based on Star generative adversarial networks (GAN), which can convert any length of music and realize the music conversion between various styles. Based on the idea of star GAN, this model uses Mel spectrum of music and random target style tags to train a general generator to generate a specified music style. It is found that the model proposed in this paper has better conversion effect between similar styles of music; that is, when Guzheng music is converted into pop and flute, the effect is the best, the average accuracy is 89.62% and 90.32%, respectively, and the cosine similarity is also 15.8 and 20.5.


---

## Body

## 1. Introduction

In recent years, the progress of Guzheng art is obvious to all. The creation and expression of Guzheng music compete with each other like a hundred flowers in full bloom. The traditional national instrument Guzheng, which has been inherited for thousands of years, has beautiful timbre and simple shape and is deeply loved by the public. The cultural heritage of the Guzheng has been developing over time. Since the 1950s, the Guzheng has entered the peak of innovation and creation of Guzheng music [1]. Especially after the 1990s, composers have compiled a variety of modern Guzheng music that conforms to the development of the times and aesthetics, breaking the limitations of the traditional Guzheng style, being all inclusive, integrating national characteristics, and boldly combining western music. At the same time, it has also made innovations in playing techniques, which has injected new vitality into the development of Guzheng. Up to now, while paying attention to preserving the characteristics of traditional Zheng music, Guzheng has been constantly absorbing, integrating, and innovating, constantly injecting new vitality, and bursting out new vitality [2].In the contemporary social background, people’s ability to adapt to the development of globalization is becoming stronger and stronger, and aesthetic changes are becoming faster and faster, so more and more new music forms emerge as the times require. Therefore, as contemporary musicians, we should treat this traditional and modern change with an inclusive heart, and we should adapt to and integrate into new music forms with our own strength. Keeping the traditional music form unchanged, we should better integrate the Guzheng with the current music form, instead of turning it into a “fast-food” performance form but should continue to develop better all the time [3]. This is the direction that every musician works together.Reviewing the current academic research results, the research on Guzheng art focuses on performance skills, aesthetic characteristics, development process, etc. At the theoretical level, there is still insufficient effort to deconstruct the musical style of important Guzheng works and sort out and summarize the performance techniques, so there is a large research space [4]. Therefore, exploring how to change the style of Guzheng music will make Guzheng music absorb constantly in different cultures, regain its vitality, and emit irreplaceable unique charm.The goal of music conversion task is to obtain new music with the overall melody or structure of the specified music and the musical characteristics of other specified tracks by learning and extracting the structure, melody, and other various information in the music of different tracks, and recombining them. Similar to image conversion tasks, music conversion can also be divided into two types: style conversion and domain conversion.The transition from music to visual style is a challenging but important cross model learning problem in creative practice. The main difference between it and the traditional image style transfer problem is that the style information is provided by music rather than images. Assuming that the music features can be correctly mapped to the visual content through the semantic link between the two domains, we solve the music to visual style conversion problem through two steps: music visualization and style conversion. The music visualization network generates an image-based music representation from music data using an encoder generator architecture with a conditional generation countermeasure network. Music style transformation refers to the transformation of a piece of music into new music with the style characteristics of another piece of music while retaining its content characteristics. Among them, the content feature of music generally refers to the overall melody or structure of music, and the style feature generally refers to the genre of music, which is usually extracted from music by pretrained convolutional neural networks (CNN). The goal of music domain conversion is to learn the overall information of music in different domains under the same type, while retaining the overall content of any music in one domain, converting it into music that sounds to belong to another domain under the same type. Among them, type refers to the type of music features that are changed in the conversion process, and domain refers to the specific type of music features.As a technology that converts music from source genre to target genre while keeping the overall content of source music unchanged, there are many possible application directions in today’s huge digital music market. For example, when a musician reproduces another song of a different style, if the two music belong to roughly the same genre, they can usually create new songs by slightly changing their playing instruments and combining their own unique sounds. However, the reality is that the music works of the two musicians usually show different genre styles. In this case, the process of manually converting music belonging to one genre into music belonging to another genre will become very complex, and the workload will become very huge. Music genre conversion can assist musicians in accelerating this music creation process, and even completely realizing automation, so that people can enjoy music that cannot be created by humans alone.In a word, as a new research field, music style transformation is expected to provide real, vivid, and creative music and become an auxiliary tool to accelerate the process of music creation. So far, generative adversarial networks (GAN) [5] have made many remarkable achievements in computer vision related fields such as random image generation [6, 7] and image-to-image conversion [8–10], but its research achievements in audio modelling and conversion are still relatively few. Choi et al. [11] proposed a model Star GAN that uses a single generator to realize unsupervised image to image conversion between unpaired image data in multiple domains. In view of the success of Star GAN in the image-to-image conversion task, Rizos et al. [12] used the same architecture to convert the Mel spectrum of speech between different emotions. Kameoka et al. [13] enhanced the additional classifier in the original Star GAN and used it to convert the Mel spectrum of speech between different speakers. So far, modelling with GAN in the audio field is still a challenge [14].Through analyzing the existing research, this paper finds that the existing research mainly realizes the one-way conversion between the two genres, and the conversion between multiple genres of music is mainly achieved by repeatedly establishing the conversion model between the two genres. This method is easy to lead to problems such as a large number of models and complex practical applications. Based on this, this paper analyzes the style of Guzheng music and the style of other music genres, studies the conversion between Guzheng music and other music, establishes a Guzheng music style conversion model based on Star GAN, and realizes the mutual conversion of Guzheng music between different genres based on retaining the overall content of Guzheng music in a given variety of genres of music.


## 2. Generation Countermeasure Network and Related Technologies

With the continuous development of neural network model and deep learning technology and theory, deep learning models such as generative countermeasure network have been widely used in image, text, audio, and other fields. The basic idea of neural network, as a common machine learning model, is to use high-dimensional nonlinear functions to estimate or approximate model the data. Deep learning is an algorithm based on neural network to learn the hidden features in data [5]. With the continuous development of deep learning theory and technology, a large number of novel deep learning models have been proposed and applied to various scenarios.


### 2.1. Mel Spectrum

Mel scale is a nonlinear scale unit. It is defined and named according to Stevens, and Ge et al. [15] found that the human ear presents an approximate logarithmic relationship with the Hertz under the frequency scale when sensing the pitch intensity in sound in 1937.(1)m=2595log101+f700.For sound signals, when the frequency is greater than 500 Hz, the human ear’s perception of pitch changes, and the frequency changes under the Mel scale are linear with the changes of Hertz under the frequency scale; that is, they are two times different from the changes of Hertz under the frequency scale. The approximate transformation between the frequencym under the Mel scale and the Hertz f under the frequency scale can be carried out through the above formula to obtain the time-frequency representation of the Mel scale spectrum as shown in Figure 1, also known as the Mel spectrum.Figure 1
Mel spectrum.


### 2.2. Griffin-Lim Algorithm

In order to effectively convert the time-frequency representation of low sampling rate obtained from time-frequency analysis of music back to time-domain waveform music, people have explored many different signal processing methods, among which Griffin-Lim algorithm [16] is the most widely used. This algorithm can effectively decode the time-frequency representation obtained from the commonly used STFT time-frequency analysis method back to the time-domain waveform through a large number of calculations [17].Because the phase information will be lost in the time-frequency analysis of music, it is not easy to reconstruct the time-domain waveform music from the time-frequency representation without phase information. In order to estimate and reconstruct the phase information from the given time-frequency representation, Griffin-Lim algorithm uses an iterative method to minimize the mean square error between the time-frequency representation of the reconstructed waveform music and the given time-frequency representation in each iteration. The Griffin-Lim algorithm flow is shown in Algorithm 1.Algorithm 1:Griffin-Lim algorithm flow
Required: time-frequency representation S, short-time Fourier transformf, inverse short-time Fourier transform f − 1.Output: waveform music signal xi reconstructed for theith time.(1)
Randomly initialize the phase of time-frequency representationP0, and waveform music signal;(2)
Repeat;(3)
Fori do;(4)
Time frequency analysis ofx0, and get a new time-frequency representation S0 and phase P0;(5)
Discard the newly obtained time-frequency representationS0, and use the original time-frequency representation S0 and the newly obtained phase P0 to reconstruct the waveform music signal;(6)
Time frequency analysis ofx1, and get a new time-frequency representation S1 and phase P1;(7)
Minimize the mean square error between the time-frequency representation of the reconstructed waveform music signal and the original time-frequency representationS;(8)
End for;(9)
Until error convergence.


### 2.3. Generative Adversarial Network

Generative adversarial network (GAN) was proposed by Goodfellow et al. [18] in 2014. Since it was proposed, it has been widely studied and is one of the hottest research topics in today’s academic circles. Yann Le, the legend of deep learning, once said in the Q&A on quota that “Gan is the most interesting and exciting idea in the field of machine learning in the past decade.”GAN is mainly inspired by the zero-sum game idea in game theory. Its basic structure is shown in Figure2, including two parts: generator (G) and discriminator (D). These two models are generally realized by neural networks, which can be constructed by deep neural networks with deep structure. In addition, they can also be realized by any form of differentiable system that can map data in space to other spaces [3]. The generator randomly samples from the latent space; that is, it takes the random noise vector as the input, and its goal is to capture and fit the distribution of the real sample and generate a data sample G(z) as similar as the real sample as possible, so as to deceive the discriminator as much as possible. The discriminator is usually a binary classifier, which takes the real samples or the data samples generated by the generator as the input, and its goal is to distinguish the generated samples from the real samples as accurately as possible. The two models constantly compete and adjust parameters, and the ultimate goal is to make the discriminator unable to accurately judge the authenticity of the generated samples, that is, whether the generated samples are real samples. The loss function of GAN in the training process is shown in the following formula:(2)minGmaxDVD,G=Ex∼PdataxlogDx+Ez∼pzzlog1−DGz,where pdata(x) represents the data distribution of real samples, and pz(z) represents the data distribution of random noise vectors. GAN learns by playing games between generator and discriminator. The optimization of model parameters is a minimum maximum optimization problem. The confrontation game between the generator and the discriminator and the adjustment of parameters end at a saddle point, which is the minimum value for the generator; that is, minimize the function V(D, G), and minimize the difference between the generated sample and the real sample. For the discriminator, which is the maximum value, that is, the maximization function V(D, G), the discriminator’s ability to distinguish between real samples and generated samples is improved as much as possible. In other words, the training and learning process of GAN is essentially a “binary minimax game” process, and its ultimate goal is to achieve Nash equilibrium [19, 20]. Therefore, it can be considered that the generator has learned the distribution of real samples and can generate enough “real” new data samples.Figure 2
Generate the basic structure of confrontation network.


### 2.4. Convolutional Neural Network

Star GAN [11] is a model designed for unsupervised multidomain image-to-image conversion tasks. Its goal is to use a generator g to learn the star mapping relationship between images in multiple domains without any paired training data, in which the numbers 1 to 5 represent image data belonging to different domains.Considering the above star topology, Star GAN uses a generatorg and a discriminator d to form a confrontation network, and its model structure and training process are shown in Figure 3. Among them, D accepts the real image and the false image (generated by the generator) as input, respectively, and predicts the domain type of the input image while judging its authenticity. G simultaneously accepts the image and the target domain label as input and outputs the converted false image.Figure 3
Star GAN model structure and training process [11].During training, Star GAN first uses the real image to traind to make it more capable of distinguishing whether the image is true or not and then trains g to convert the image from the source domain to the target domain and from the target domain back to the source domain, respectively, so that the converted false image can deceive D. The loss function used includes Wasserstein Gan objective with gradient penalty (WGAN-GP) [21], which can make the training process of Gan more stable.(3)Ladv=ExDx−Ex,cDGx,c−λgpEx⌢∇x⌢Dx⌢2−12,where x represents the input image, x⌢ represents the image evenly sampled between a pair of real and generated images, c represents the target domain label, and λgp is set to 10 in the experiment.And two classification losses were used to predict the category of the real and converted images, respectively [11]:(4)Lclsr=Ex,c−logDclscx,Lclsf=Ex,c−logDclscGx,c,where c′ represents the source domain label. And a reconstruction loss based on cyclic consistency loss for constraining the model to retain the original input content during the conversion process [11]is used:(5)Lrec=Ex,c,cx−GGx,c1.


## 2.1. Mel Spectrum

Mel scale is a nonlinear scale unit. It is defined and named according to Stevens, and Ge et al. [15] found that the human ear presents an approximate logarithmic relationship with the Hertz under the frequency scale when sensing the pitch intensity in sound in 1937.(1)m=2595log101+f700.For sound signals, when the frequency is greater than 500 Hz, the human ear’s perception of pitch changes, and the frequency changes under the Mel scale are linear with the changes of Hertz under the frequency scale; that is, they are two times different from the changes of Hertz under the frequency scale. The approximate transformation between the frequencym under the Mel scale and the Hertz f under the frequency scale can be carried out through the above formula to obtain the time-frequency representation of the Mel scale spectrum as shown in Figure 1, also known as the Mel spectrum.Figure 1
Mel spectrum.


## 2.2. Griffin-Lim Algorithm

In order to effectively convert the time-frequency representation of low sampling rate obtained from time-frequency analysis of music back to time-domain waveform music, people have explored many different signal processing methods, among which Griffin-Lim algorithm [16] is the most widely used. This algorithm can effectively decode the time-frequency representation obtained from the commonly used STFT time-frequency analysis method back to the time-domain waveform through a large number of calculations [17].Because the phase information will be lost in the time-frequency analysis of music, it is not easy to reconstruct the time-domain waveform music from the time-frequency representation without phase information. In order to estimate and reconstruct the phase information from the given time-frequency representation, Griffin-Lim algorithm uses an iterative method to minimize the mean square error between the time-frequency representation of the reconstructed waveform music and the given time-frequency representation in each iteration. The Griffin-Lim algorithm flow is shown in Algorithm 1.Algorithm 1:Griffin-Lim algorithm flow
Required: time-frequency representation S, short-time Fourier transformf, inverse short-time Fourier transform f − 1.Output: waveform music signal xi reconstructed for theith time.(1)
Randomly initialize the phase of time-frequency representationP0, and waveform music signal;(2)
Repeat;(3)
Fori do;(4)
Time frequency analysis ofx0, and get a new time-frequency representation S0 and phase P0;(5)
Discard the newly obtained time-frequency representationS0, and use the original time-frequency representation S0 and the newly obtained phase P0 to reconstruct the waveform music signal;(6)
Time frequency analysis ofx1, and get a new time-frequency representation S1 and phase P1;(7)
Minimize the mean square error between the time-frequency representation of the reconstructed waveform music signal and the original time-frequency representationS;(8)
End for;(9)
Until error convergence.


## 2.3. Generative Adversarial Network

Generative adversarial network (GAN) was proposed by Goodfellow et al. [18] in 2014. Since it was proposed, it has been widely studied and is one of the hottest research topics in today’s academic circles. Yann Le, the legend of deep learning, once said in the Q&A on quota that “Gan is the most interesting and exciting idea in the field of machine learning in the past decade.”GAN is mainly inspired by the zero-sum game idea in game theory. Its basic structure is shown in Figure2, including two parts: generator (G) and discriminator (D). These two models are generally realized by neural networks, which can be constructed by deep neural networks with deep structure. In addition, they can also be realized by any form of differentiable system that can map data in space to other spaces [3]. The generator randomly samples from the latent space; that is, it takes the random noise vector as the input, and its goal is to capture and fit the distribution of the real sample and generate a data sample G(z) as similar as the real sample as possible, so as to deceive the discriminator as much as possible. The discriminator is usually a binary classifier, which takes the real samples or the data samples generated by the generator as the input, and its goal is to distinguish the generated samples from the real samples as accurately as possible. The two models constantly compete and adjust parameters, and the ultimate goal is to make the discriminator unable to accurately judge the authenticity of the generated samples, that is, whether the generated samples are real samples. The loss function of GAN in the training process is shown in the following formula:(2)minGmaxDVD,G=Ex∼PdataxlogDx+Ez∼pzzlog1−DGz,where pdata(x) represents the data distribution of real samples, and pz(z) represents the data distribution of random noise vectors. GAN learns by playing games between generator and discriminator. The optimization of model parameters is a minimum maximum optimization problem. The confrontation game between the generator and the discriminator and the adjustment of parameters end at a saddle point, which is the minimum value for the generator; that is, minimize the function V(D, G), and minimize the difference between the generated sample and the real sample. For the discriminator, which is the maximum value, that is, the maximization function V(D, G), the discriminator’s ability to distinguish between real samples and generated samples is improved as much as possible. In other words, the training and learning process of GAN is essentially a “binary minimax game” process, and its ultimate goal is to achieve Nash equilibrium [19, 20]. Therefore, it can be considered that the generator has learned the distribution of real samples and can generate enough “real” new data samples.Figure 2
Generate the basic structure of confrontation network.


## 2.4. Convolutional Neural Network

Star GAN [11] is a model designed for unsupervised multidomain image-to-image conversion tasks. Its goal is to use a generator g to learn the star mapping relationship between images in multiple domains without any paired training data, in which the numbers 1 to 5 represent image data belonging to different domains.Considering the above star topology, Star GAN uses a generatorg and a discriminator d to form a confrontation network, and its model structure and training process are shown in Figure 3. Among them, D accepts the real image and the false image (generated by the generator) as input, respectively, and predicts the domain type of the input image while judging its authenticity. G simultaneously accepts the image and the target domain label as input and outputs the converted false image.Figure 3
Star GAN model structure and training process [11].During training, Star GAN first uses the real image to traind to make it more capable of distinguishing whether the image is true or not and then trains g to convert the image from the source domain to the target domain and from the target domain back to the source domain, respectively, so that the converted false image can deceive D. The loss function used includes Wasserstein Gan objective with gradient penalty (WGAN-GP) [21], which can make the training process of Gan more stable.(3)Ladv=ExDx−Ex,cDGx,c−λgpEx⌢∇x⌢Dx⌢2−12,where x represents the input image, x⌢ represents the image evenly sampled between a pair of real and generated images, c represents the target domain label, and λgp is set to 10 in the experiment.And two classification losses were used to predict the category of the real and converted images, respectively [11]:(4)Lclsr=Ex,c−logDclscx,Lclsf=Ex,c−logDclscGx,c,where c′ represents the source domain label. And a reconstruction loss based on cyclic consistency loss for constraining the model to retain the original input content during the conversion process [11]is used:(5)Lrec=Ex,c,cx−GGx,c1.


## 3. Guzheng Music Translation Based on Star Gan

There are two kinds of Guzheng music conversion, one-to-one music conversion and one-to-many music style conversions. One-to-one music conversion refers to the conversion of Guzheng music into another kind of music, while retaining the content of the conversion of Guzheng music. The goal of one-to-many music style conversion is to realize the mutual conversion between Guzheng music and mixed music, so that the content of the converted music is as close to Guzheng music as possible, and the style it belongs to should be as close to the music style it actually wants to convert to, also known as multimusic style conversion.In order to solve the above problems as much as possible, this paper proposes a Guzheng music translation based on Star GAN (GZ-Star GAN) by learning the existing music style and instrument conversion models and generating a confrontation network model, which can convert any length of Guzheng music into a variety of target styles. Different from the original Star GAN network, this paper improves the network structure and loss function in GZ-Star GAN to improve the stability of the model during training, the retention effect of the overall content in Guzheng music, and the conversion effect between target music.In GZ-Star GAN, first, based on the idea of Star GAN, this paper uses the Mel spectrum of music and random target style tags to train a general generator to generate Mel spectrum of the specified style and trains a discriminator to judge the authenticity and style type of Mel spectrum at the same time, so as to realize the style conversion. Secondly, the twin network is introduced to replace the cyclic consistency loss in the original Star GAN to constrain the eigenvectors of the Mel spectrum before and after the conversion, so as to make them equal as much as possible, so as to retain the overall content of the Guzheng music. Third, the differences between music styles are included in the loss function calculation as a weight, and the styles with small differences are given a larger weight value to improve the conversion effect between similar style music. At the same time, in order to improve the stability of the model during training, the hinge loss is introduced to replace the WGAN-GP counter loss in the original Star Gan, and the spectrum normalization operation is adopted for all convolution kernels in the generator and discriminator.The structure of Guzheng music style conversion model GZ-Star GAN based on Star GAN is shown in Figure4. It is composed of four parts: time-frequency analysis module, style conversion network, content retention network, and waveform reconstruction module. Among them, the time-frequency analysis module extracts the time-frequency representation of logarithmic amplitude Mel spectrum of time-domain waveform music through short-time Fourier transform and Mel filter. The music style conversion network includes a generator and a discriminator for performing style conversion on the time-frequency representation of the Mel spectrum of music. The content retention network is used to retain the overall content of Guzheng music in the process of style conversion. The waveform reconstruction module reconstructs the converted Mel spectrum to echo music through Griffin Lim algorithm.Figure 4
GZ-Star GAN model structure.


### 3.1. Transformation Network

Based on the idea of the domain image to image conversion model star Gan, this paper constructs a style conversion network in GZ-Star Gan to realize the mutual conversion between Guzheng music and other music styles. The conversion network is composed of a generator and a discriminator. The generator’s goal is to convert the Mel spectrum of Guzheng music into a specified target style. The discriminator’s goal is to be able to distinguish the real Mel spectrum from the Mel spectrum converted by the generator and to identify the style of the Mel spectrum.For the generator, GZ-Star Gan uses subpixel convolution network [22], instead of deconvolution network, and upsamples the Mel spectrum on the basis of the original Star Gan generator architecture, to prevent the deconvolution network used by the original Star Gan generator from generating chessboard artifacts in the converted Mel spectrum, thus causing serious noise in the reconstructed waveform music. For the discriminator, GZ-Star Gan refers to the network architecture of the discriminator in Star Gan and adds a network layer for classifying various styles on the basis of the standard discriminator structure to determine whether the Mel spectra of various styles converted by the generator belong to the correct target style.For a given original Mel spectrum, in order to make the Mel spectrum converted by the generator real enough, the generator and the discriminator need to be trained at the same time. Because the WGAN-GP used by Star GAN uses Wasserstein distance to measure the difference between the two distributions, it will show the characteristics that the error is proportional to the convergence speed during training, and it is not suitable for tasks involving many different types of data and needs to maintain the relative balance of the overall effect.Therefore, in order to make the confrontation between the generator and the discriminator, the learning process can be more stable. On the one hand, drawing on the recent research progress in the field of GAN, in GZ-Star GAN, spectral normalization (SN) [23] is used for all convolution cores in convolution neural networks used in generators and discriminators; that is, the network parameters of convolution neural networks at each layer are divided by the spectral norm of the network parameters at that layer, replacing the Wasserstein distance used in WGAN-GP networks as a method to realize Lipschitz constraint in the training process of the model. The calculation process of realizing Lipschitz constraint is shown as(6)∇xfx2=DNWNσWN…D1W1σW12≤∏i=1Nσwiσwi=1,where f (x) is the N-layer neural network with x as the input, ∇xfx2 represents the gradient of f (x), DN is the diagonal matrix used to represent the action of the nonlinear activation function RELU of the N-layer network, the positive and negative of its diagonal elements are consistent with the input, WN is the parameter matrix of the N -layer network, and σWN is the maximum singular value of the parameter matrix WN.


### 3.2. Content Retention Network

Aiming at the problem that the original Star Gan uses the cyclic consistency loss to retain the original input content during the conversion process, it is easy to introduce additional noise outside the content, thus affecting the retention effect of the Guzheng music content. GZ-Star GAN introduced a content retention network outside the style conversion network to achieve the research goal of converting music between multiple styles and preserving the overall content of Guzheng music as much as possible. In order to realize the retention of the whole content of Guzheng music in the process of style conversion, this paper, based on the idea of double network, constrains the Mel spectrum feature vectors before and after the conversion to be equal as much as possible.First, in GZ-Star GAN, the transformation vector between paired Mel spectrum sets (x1, x2) is defined as(7)tx1,x2=x1−x2.During training, the content preserving networkS extracts and encodes the content feature information contained in the paired Mel spectrum set before conversion and the paired Mel spectrum set converted by the generator G. The training goal is to make the Mel spectrum set after conversion retain the overall music content in the Mel spectrum set before conversion as much as possible, even if the conversion vectors defined in formula (7) meet the relationship shown in formula (8) as much as possible, where S(x) represents the output vector of the network with the combined x of the segmented Mel spectrum as the input, and the content is reserved.(8)tSM˜1,M˜2=tSM˜^1″,M˜^2″.In the training process, referring to the conversion loss originally proposed by Amodio et al. [24], the type conversion loss function is used to optimize the image-to-image conversion task. It should be noted that, in the training process, the loss of genre conversion is optimized by G and S networks together to achieve the goal of reducing iteration, which makes it easy for S to learn a simple identity map to meet the requirements. For example, for any input, s outputs a fixed value. Therefore, a standard twin boundary contrast loss is added here for the content retention networks [25, 26] to avoid the above situation. Among them, δ is a fixed value, set to 2 in this article. By using this loss, the twin network can make the distance between any two vectors obtained by the input coding at least different and avoid the network learning becoming a simple identity mapping function.


### 3.3. Dynamic Weight of Loss Function

Aiming at the problem that the existing models tend to learn the conversion between style music with obvious differences and that the conversion effect between similar style music is poor, this paper proposes a dynamic weight scheme of loss function, which takes the differences between music styles as weights into the calculation of loss function and gives a larger weight value to the styles with small differences, so as to effectively increase the loss between similar style music and improve the conversion effect of the model [16]. The whole scheme includes two parts: calculating the difference between different styles of music and calculating the dynamic weight of loss function based on the difference between styles of music [27]. Its basic process is shown in Figure 5.Figure 5
Loss item dynamic weighting scheme.Among them,i and j represent the original Mel spectrum of Guzheng and other styles randomly selected from the original Mel spectrum collection. C represents the additional classifier, which represents the style feature vector extracted by the additional classifier respectively, and C(i) and C(j) represent the dynamic weight of the loss function calculated based on the difference between Guzheng and other music [28].


## 3.1. Transformation Network

Based on the idea of the domain image to image conversion model star Gan, this paper constructs a style conversion network in GZ-Star Gan to realize the mutual conversion between Guzheng music and other music styles. The conversion network is composed of a generator and a discriminator. The generator’s goal is to convert the Mel spectrum of Guzheng music into a specified target style. The discriminator’s goal is to be able to distinguish the real Mel spectrum from the Mel spectrum converted by the generator and to identify the style of the Mel spectrum.For the generator, GZ-Star Gan uses subpixel convolution network [22], instead of deconvolution network, and upsamples the Mel spectrum on the basis of the original Star Gan generator architecture, to prevent the deconvolution network used by the original Star Gan generator from generating chessboard artifacts in the converted Mel spectrum, thus causing serious noise in the reconstructed waveform music. For the discriminator, GZ-Star Gan refers to the network architecture of the discriminator in Star Gan and adds a network layer for classifying various styles on the basis of the standard discriminator structure to determine whether the Mel spectra of various styles converted by the generator belong to the correct target style.For a given original Mel spectrum, in order to make the Mel spectrum converted by the generator real enough, the generator and the discriminator need to be trained at the same time. Because the WGAN-GP used by Star GAN uses Wasserstein distance to measure the difference between the two distributions, it will show the characteristics that the error is proportional to the convergence speed during training, and it is not suitable for tasks involving many different types of data and needs to maintain the relative balance of the overall effect.Therefore, in order to make the confrontation between the generator and the discriminator, the learning process can be more stable. On the one hand, drawing on the recent research progress in the field of GAN, in GZ-Star GAN, spectral normalization (SN) [23] is used for all convolution cores in convolution neural networks used in generators and discriminators; that is, the network parameters of convolution neural networks at each layer are divided by the spectral norm of the network parameters at that layer, replacing the Wasserstein distance used in WGAN-GP networks as a method to realize Lipschitz constraint in the training process of the model. The calculation process of realizing Lipschitz constraint is shown as(6)∇xfx2=DNWNσWN…D1W1σW12≤∏i=1Nσwiσwi=1,where f (x) is the N-layer neural network with x as the input, ∇xfx2 represents the gradient of f (x), DN is the diagonal matrix used to represent the action of the nonlinear activation function RELU of the N-layer network, the positive and negative of its diagonal elements are consistent with the input, WN is the parameter matrix of the N -layer network, and σWN is the maximum singular value of the parameter matrix WN.


## 3.2. Content Retention Network

Aiming at the problem that the original Star Gan uses the cyclic consistency loss to retain the original input content during the conversion process, it is easy to introduce additional noise outside the content, thus affecting the retention effect of the Guzheng music content. GZ-Star GAN introduced a content retention network outside the style conversion network to achieve the research goal of converting music between multiple styles and preserving the overall content of Guzheng music as much as possible. In order to realize the retention of the whole content of Guzheng music in the process of style conversion, this paper, based on the idea of double network, constrains the Mel spectrum feature vectors before and after the conversion to be equal as much as possible.First, in GZ-Star GAN, the transformation vector between paired Mel spectrum sets (x1, x2) is defined as(7)tx1,x2=x1−x2.During training, the content preserving networkS extracts and encodes the content feature information contained in the paired Mel spectrum set before conversion and the paired Mel spectrum set converted by the generator G. The training goal is to make the Mel spectrum set after conversion retain the overall music content in the Mel spectrum set before conversion as much as possible, even if the conversion vectors defined in formula (7) meet the relationship shown in formula (8) as much as possible, where S(x) represents the output vector of the network with the combined x of the segmented Mel spectrum as the input, and the content is reserved.(8)tSM˜1,M˜2=tSM˜^1″,M˜^2″.In the training process, referring to the conversion loss originally proposed by Amodio et al. [24], the type conversion loss function is used to optimize the image-to-image conversion task. It should be noted that, in the training process, the loss of genre conversion is optimized by G and S networks together to achieve the goal of reducing iteration, which makes it easy for S to learn a simple identity map to meet the requirements. For example, for any input, s outputs a fixed value. Therefore, a standard twin boundary contrast loss is added here for the content retention networks [25, 26] to avoid the above situation. Among them, δ is a fixed value, set to 2 in this article. By using this loss, the twin network can make the distance between any two vectors obtained by the input coding at least different and avoid the network learning becoming a simple identity mapping function.


## 3.3. Dynamic Weight of Loss Function

Aiming at the problem that the existing models tend to learn the conversion between style music with obvious differences and that the conversion effect between similar style music is poor, this paper proposes a dynamic weight scheme of loss function, which takes the differences between music styles as weights into the calculation of loss function and gives a larger weight value to the styles with small differences, so as to effectively increase the loss between similar style music and improve the conversion effect of the model [16]. The whole scheme includes two parts: calculating the difference between different styles of music and calculating the dynamic weight of loss function based on the difference between styles of music [27]. Its basic process is shown in Figure 5.Figure 5
Loss item dynamic weighting scheme.Among them,i and j represent the original Mel spectrum of Guzheng and other styles randomly selected from the original Mel spectrum collection. C represents the additional classifier, which represents the style feature vector extracted by the additional classifier respectively, and C(i) and C(j) represent the dynamic weight of the loss function calculated based on the difference between Guzheng and other music [28].


## 4. Music Conversion Experiment Based on GZ-Star GAN

### 4.1. Dataset

In order to evaluate the effectiveness of GZ-Star GAN model in the task of music style conversion and its flexibility in other types of music data, this paper uses GZ-Star GAN to carry out music style conversion experiments on GTZAN style music data set and music instrument conversion experiments on 415-Chinese traditional instrument (CTI) instrument music data set.GTZAN data set is often used for waveform music style classification task, which contains 10 styles, each style is composed of 100 music samples, and the duration of each music sample is 30 seconds, a total of 1000 music samples. Each music sample in the dataset is a 22050 Hz mono 16 bit audio file in wav format. 415-CTI data set is collected and sorted out from various sources such as CD, microphone recording, and Internet according to the types of traditional Chinese musical instruments. It contains 12 kinds of traditional Chinese musical instruments, each of which is also composed of 100 music samples with a length of 30 s, a total of 1200 music samples [29, 30]. Each music sample in the dataset is a 44100 Hz mono 16 bit audio file in wav format.The research goal of this paper is to realize the mutual conversion between various styles of music. Therefore, considering the calculation cost of the model and the representativeness of the selected styles and instruments, the music of blues, classical, pop, and jazz, as well as the music of suona, pipa, flute, and erhu, are selected from GTZAN and 415-CTI data sets for experiments, The data set is divided according to the ratio of training and testing 9 : 1. Finally, the number of music samples in the training set of the two data sets is 360, and the number of music samples in the test set is 40.


### 4.2. Experimental Results and Analysis

In order to evaluate whether the proposed GZ-Star GAN model really learned to convert music of different styles or instruments, this paper uses accuracy as the evaluation index. Accuracy refers to the probability that samples are recognized correctly, involving four concepts: true positive (TP) representing the number of positive samples recognized as positive cases, false positive (FP) representing the number of negative samples recognized as positive cases, true negative (TN) representing the number of positive samples recognized as negative samples, and false negative (FN) representing the number of negative samples recognized as negative cases. Its calculation formula is(9)Accuracy=TP+TNTP+FN+TN+FP.In order to evaluate the conversion effect of musical instruments and styles of GZ-Star GAN model, it is applied to different music data sets to try to convert music between different instruments. Different music data sets are used to train the additional classifiers. The average accuracy and average cosine similarity of the trained additional classifiers on the corresponding test set are shown in Figure6, where the value range of cosine similarity is [−1, 1]. The larger the value, the more similar the two genres of music. The smaller the value, the more dissimilar the two genres of music are. Figure 6 shows that the average accuracy between Guzheng and pop is 89.62%, indicating that the two kinds of music have higher similarities in some aspects. The average accuracy between Guzheng and jazz music is the lowest, 50.3%. At the same time, Figure 6 shows that, compared with other music, the cosine similarity between Guzheng and pop music is the smallest, which also shows that the two kinds of music have higher similarity.Figure 6
Test average accuracy and cosine similarity between different music.Similarly, the instrument music data set is used to train the additional classifier, and its average accuracy and average cosine similarity on the corresponding test set are shown in Figure7. As can be seen from Figure 7, the average accuracy between Guzheng and flute music is 90.32%, indicating that the two instruments have higher similarities in some aspects. The average accuracy between Guzheng and Suona music is the lowest, 50.3%. At the same time, Figure 7 shows that, compared with other instruments, the cosine similarity between Guzheng and flute music is the smallest, which also shows that the musical similarity of these two instruments is higher.Figure 7
Test average accuracy and cosine similarity between different instruments.


## 4.1. Dataset

In order to evaluate the effectiveness of GZ-Star GAN model in the task of music style conversion and its flexibility in other types of music data, this paper uses GZ-Star GAN to carry out music style conversion experiments on GTZAN style music data set and music instrument conversion experiments on 415-Chinese traditional instrument (CTI) instrument music data set.GTZAN data set is often used for waveform music style classification task, which contains 10 styles, each style is composed of 100 music samples, and the duration of each music sample is 30 seconds, a total of 1000 music samples. Each music sample in the dataset is a 22050 Hz mono 16 bit audio file in wav format. 415-CTI data set is collected and sorted out from various sources such as CD, microphone recording, and Internet according to the types of traditional Chinese musical instruments. It contains 12 kinds of traditional Chinese musical instruments, each of which is also composed of 100 music samples with a length of 30 s, a total of 1200 music samples [29, 30]. Each music sample in the dataset is a 44100 Hz mono 16 bit audio file in wav format.The research goal of this paper is to realize the mutual conversion between various styles of music. Therefore, considering the calculation cost of the model and the representativeness of the selected styles and instruments, the music of blues, classical, pop, and jazz, as well as the music of suona, pipa, flute, and erhu, are selected from GTZAN and 415-CTI data sets for experiments, The data set is divided according to the ratio of training and testing 9 : 1. Finally, the number of music samples in the training set of the two data sets is 360, and the number of music samples in the test set is 40.


## 4.2. Experimental Results and Analysis

In order to evaluate whether the proposed GZ-Star GAN model really learned to convert music of different styles or instruments, this paper uses accuracy as the evaluation index. Accuracy refers to the probability that samples are recognized correctly, involving four concepts: true positive (TP) representing the number of positive samples recognized as positive cases, false positive (FP) representing the number of negative samples recognized as positive cases, true negative (TN) representing the number of positive samples recognized as negative samples, and false negative (FN) representing the number of negative samples recognized as negative cases. Its calculation formula is(9)Accuracy=TP+TNTP+FN+TN+FP.In order to evaluate the conversion effect of musical instruments and styles of GZ-Star GAN model, it is applied to different music data sets to try to convert music between different instruments. Different music data sets are used to train the additional classifiers. The average accuracy and average cosine similarity of the trained additional classifiers on the corresponding test set are shown in Figure6, where the value range of cosine similarity is [−1, 1]. The larger the value, the more similar the two genres of music. The smaller the value, the more dissimilar the two genres of music are. Figure 6 shows that the average accuracy between Guzheng and pop is 89.62%, indicating that the two kinds of music have higher similarities in some aspects. The average accuracy between Guzheng and jazz music is the lowest, 50.3%. At the same time, Figure 6 shows that, compared with other music, the cosine similarity between Guzheng and pop music is the smallest, which also shows that the two kinds of music have higher similarity.Figure 6
Test average accuracy and cosine similarity between different music.Similarly, the instrument music data set is used to train the additional classifier, and its average accuracy and average cosine similarity on the corresponding test set are shown in Figure7. As can be seen from Figure 7, the average accuracy between Guzheng and flute music is 90.32%, indicating that the two instruments have higher similarities in some aspects. The average accuracy between Guzheng and Suona music is the lowest, 50.3%. At the same time, Figure 7 shows that, compared with other instruments, the cosine similarity between Guzheng and flute music is the smallest, which also shows that the musical similarity of these two instruments is higher.Figure 7
Test average accuracy and cosine similarity between different instruments.


## 5. Discussion and Conclusion

Aiming at the conversion between Guzheng music and other music, this paper proposes a Guzheng music conversion model based on Star GAN, which can convert any length of music and realize the music conversion between various styles. The main work of this paper includes the following aspects:(1)
This paper presents a Guzheng music conversion model GZ-Star GAN. Based on the idea of Star GAN, this model uses the Mel spectrum of music and random target style tags to train a general generator to generate Mel spectrum of a specified style and train a discriminator to judge the authenticity and style type of Mel spectrum at the same time, which can convert Guzheng music of any length into other target music and realize the conversion of Guzheng music.(2)
In order to improve the retention effect of Guzheng music content in the process of music conversion, a content retention network is introduced to constrain the eigenvectors of Mel spectrum before and after conversion, making them equal as much as possible, which improves the retention effect of the overall content of Guzheng music.(3)
In order to improve the conversion effect between similar music in the process of music conversion, this paper adds the difference between music as a weight to a dynamic weighting scheme of loss function, and the music with small difference is given a larger weight value, which effectively increases the loss between similar music and improves the conversion effect of the model between similar music.(4)
In order to verify the music conversion effect of the model, experiments are carried out on the style and instrument music data sets, respectively. The experimental results show that the model proposed in this paper has a good conversion effect between similar styles of music and also has a good effect between other styles of music. When Guzheng music is converted into pop music and flute music, the effect is the best, the average accuracy reaches 89.62% and 90.32%, respectively, and the cosine similarity also reaches 15.8 and 20.5. It verifies the flexibility that this model can be applied to other types of music styles.



---

*Source: 4950231-2022-09-22.xml*