# Development of the Music Teaching System Based on Speech Recognition and Artificial Intelligence

Authors: Zhang Zhichao
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Intelligent music teaching is the direction of subsequent reforms in music teaching methods. In order to improve the intelligence of music teaching, this paper conducts research on speech recognition technology. The speech conversion system based on Multiscale Star GAN extracts different levels of multiscale features of the global features of music utterance through the multiscale structure, which enhances the details of the converted speech and uses residual connections to alleviate the problem of gradient disappearance and enable the network to spread more deeply. In addition, after improving the speech recognition algorithm, this paper combines the needs of music teaching to construct a music teaching system based on speech recognition and artificial intelligence and designs system functional modules. Finally, this paper evaluates the performance of the system constructed in this paper by means of teaching experiments. From the experimental analysis, it can be seen that the system constructed in this paper has a good teaching effect.


---

## Body

## 1. Introduction

All-round development is the ideal of education that human beings strive to pursue, and music education contains a splendid style and long-standing characteristics that can help the educated to get closer to all-round development [1]. Therefore, we must make efforts to build a close connection between music and life in the process of learning music. Moreover, it is conducive to exploring and perfecting the auxiliary functions of music education and expanding the quality of life of young people in the music teaching classroom.The impact of numerous music cultures, local government policy support, the employment of different versions of music appreciation textbooks, and the school’s right knowledge of music education all give directions and routes for the execution of auxiliary functions of music education. Simultaneously, the employment of “Internet +” and multimedia technologies in the classroom, under the complete stimulation of the information age, gives practical feasibility for the execution of today’s music discipline’s core literacy. As a result, the present scenario makes it possible to implement the auxiliary role of music education [2].At the same time, the music education function is founded on the application of the music discipline’s core literacy against a foundation of core literacy. The auxiliary purpose of music education is to carry out the inheritance of music culture while also serving as an intermediate via its aesthetic function. It is possible for links to function. Music education can only develop the functions of moral education, intellectual education, and sports and can properly execute the essential features of music discipline, if it is founded on the aesthetic function of music. In this sense, educational concept renewal can promote the development and deepening of our music education’s auxiliary functions, contribute to the cultivation of people with a holistic development of morality, intelligence, and physical beauty, and enable the implementation of auxiliary functions of music education in teaching.The long-term goal of the auxiliary function of music education is to mold the current students into people with core music literacy. Each lesson of music teaching needs to set a staged goal, which is based on the content of the textbook and the physical and mental characteristics of the students. Staged goals can serve long-term goals and must be operable. Therefore, in addition to paying attention to the requirements of long-term goals, we must also pay attention to the unity, continuity, and gradual nature of the goals at each stage. Music education has the ability to stimulate wisdom, cultivate emotions, shape individuality, and realize human spiritual communication and integration. As an important part of general quality education, it is necessary to earnestly implement the spirit of relevant central documents, to comprehensively cultivate and develop ideal, ethical, educated, and disciplined socialist practical talents; moreover, it is necessary to fully develop the assistance of music education function to promote the cultivation of the comprehensive quality of ordinary students [3].This paper uses speech recognition technology and artificial intelligence technology to study the development of music teaching system to further improve the reliability and stability of music teaching and to provide a theoretical basis for subsequent music teaching.


## 2. Related Work

There are different perspectives on the classification of speech conversion technology. Among them, a classification angle is to divide the speech conversion method into parallel text method and nonparallel text method according to whether the conversion model requires the same text or phoneme set. Parallel text refers to speeches with the same semantics but different speaker identities, while nonparallel text refers to speeches with different semantics and different speaker identities. Another classification angle is based on the language situation of the source and target speakers, which can be divided into same-language conversion and cross-language conversion [4]. Furthermore, depending on the number of speakers of the source and target speeches throughout the training process, speech conversion techniques may be split into one-to-one, many-to-one, and many-to-many speech conversion. Alternatively, it is separated into speech conversion in the training data set and speech conversion outside the training data set, depending on whether the speaker appears in the training data set. The majority of conventional speech conversion techniques are classified as speech conversion with parallel text, which means that the parallel text is utilised as training data to develop the mapping function or conversion model. Collecting parallel speeches, on the other hand, is time-consuming and labor-intensive in real-world circumstances, and it is not accessible at all in the fields of cross-language conversion and medical help. Furthermore, even if such parallel data is gathered, most speech conversion algorithms still need the training data to be time-aligned. To overcome the issue of temporal alignment problems, the alignment method always adds mistakes, necessitating more sophisticated processes such as exact corpus preparation or human correction [5].Because of the problems with parallel text-to-speech conversion technology, nonparallel speech conversion technology has increasingly become the current standard. Some researchers have recommended that the model be assisted by background speech data [6] or a speech recognition module [7] to enable nonparallel speech conversion. Although background data and auxiliary modules aid in the resolution of nonparallel issues, they add to the complexity of the system and depend on auxiliary data. With the module’s overall performance, RBMs [8], variational autoencoders [9], and GANs are examples of probabilistic depth generation models that have been brought into the area of speech conversion in recent years [10]. These technologies do not rely on parallel data and the system performance is comparable to that of parallel speech conversion methods. The performance is equivalent. Among them, the voice conversion based on the cyclic uniform generation confrontation network [11] combines Cycle GAN [12] and gated convolutional neural networks (Gated CNNs) [13] to achieve voice conversion and obtain better experimental results. Although Cycle GAN-VC solves the dependence of the voice conversion method on parallel data and the voice quality and similarity after conversion are better, it still cannot achieve many-to-many voice conversion. Traditional voice conversion methods are mostly designed to learn one-to-one mapping, but it is very challenging to realize many-to-many voice conversion under nonparallel text. In order to solve the problem that Cycle GAN-VC cannot realize many-to-many speech conversion, the speech conversion based on star generation confrontation network (STARGAN-VC) [14] method introduces the speaker tag on the basis of Cycle GAN-VC. Realize nonparallel many-to-many speech conversion. Experiments show that the performance of this method is better than other current methods, such as many speech conversion methods based on VAE or other GANs [15].Voice conversion methods based on VAE and GAN may directly avoid text alignment issues and give a new foundation for voice conversion in nonparallel text situations. Not only is the input feature encoded into a latent space feature in the VAE-based speech conversion series model, but the speaker is also encoded into a label feature. The aforementioned two characteristics are joined in the generating step to recreate the spectral feature. The most prevalent are the auxiliary classification variational autoencoder voice conversion (Auxiliary Classifier VAE VC, ACVAE-VC) [16], cyclic variational autoencoder voice conversion (Cycle VAE VC, Cycle VAE-VC) [17], and so on. In brief, the VAE-based model meets the criteria for parallel data and can do nonparallel conversion jobs; nevertheless, the output of the model’s decoder is too smooth, resulting in conversion in the speech conversion task. The post-voice quality is poor, and there is a buzzing sound, indicating that there are issues with the converted spectrum’s naturalness and resemblance. As a result, the voice conversion system based on the VAE framework does not function well. Through the learning of generators and discriminators, the GAN-based speech conversion system may improve the quality of synthesised speech. Literature [18] increased voice quality by proposing Cycle GAN-VC2, which incorporated a two-step counter loss to further improve spectrum generation quality. Literature [19] improved and improved the original discriminator, with the capacity to extract features; literature [20] discovered that, in nonparallel data, the cross relationship between two speakers is utilised to perform speech conversion, but the resulting speech quality still has some pronunciation ambiguity and noise difficulties, the same as the above two. The same Cycle GAN-based approaches can only convert one-to-one, and there are still significant restrictions in practical applications. As a result, research into many-to-many nonparallel speech conversion has higher theoretical and practical utility.


## 3. Many-to-Many Speech Conversion Based on Star GAN under Nonparallel Text Conditions

Because Cycle GAN does not separate speaker identity features, it can only achieve one-to-one nonparallel speech conversion. If we want to achieve many-to-many, we can only train models with multiple speaker pairs. The Star GAN model was originally used on images to achieve many-to-many image conversion. Star GAN-VC separates speaker identity features on the basis of Cycle GAN and introduces classification loss to determine which speaker the generated speech belongs to and realizes many-to-many nonparallel speech conversion [21].The Star GAN is composed of a generatorG, a discriminator D, and a classifier C. The generator G is composed of an encoding network and a decoding network. Unlike the Cycle GAN, the Star GAN uses only one generator G. The network structure diagram of Star GAN is shown in Figure 1.Figure 1
Structure diagram of the Star GAN.As can be seen from Figure1, Star GAN adds separately coded target label features to guide the generated target speaker’s speech. The generator G encodes the input speech features of the source speaker to obtain the features on the semantic level and then inputs the tag codes of different speakers to decode and generate different speaker speech features. The generated features of the target speaker and the label of the source speaker are input into the generator G again to obtain the reconstructed feature of the source speaker, and the loop loss LcycG is calculated. In addition, the source speaker’s characteristics and the source speaker’s label are input into the generator G to directly generate the source speaker’s speech features, and the feature mapping loss LidG is calculated [22].(1)LcycG=Exs∼pxs|cs,cs∼pcs,ct∼pctGGxs,ct,cs−xs1.Among them,xs∼pxs|cs represents the frequency spectrum characteristics of the source speaker, cs∼pcs represents the label of the source speaker, ct∼pct represents the label of the target speaker, GGxs,ct,cs represents the spectrum characteristics of the reconstructed source speaker, and Exs∼pxs|cs,cs∼pcs,ct∼pct⋅ is the expected loss of the reconstructed source speaker’s spectrum and the true source speaker’s spectrum.(2)LidG=Ecs∼pcs,xs∼pxs|csGxs,cs−xs1.Among them,Gxs,cs is the source speaker’s spectrum characteristics obtained after the source speaker’s frequency spectrum and source speaker label are input to the generator, and Ecs∼pcs,xs∼pxs|cs⋅ is the expected loss of xs and Gxs,cs.The discriminatorD is used to judge whether the generated features conform to the feature distribution of the real speaker’s speech. The real speech and the label input of the speech speaker are input, and the probability that the speech feature conforms to the speech feature distribution of the real speaker is output. The pretraining generator distinguishes real speech features and generated speech features and minimizes the generator’s confrontation loss function LadvDG. The posttraining generator enables the generated speech features to deceive the discriminator into real speech and minimizes the generator’s confrontation loss function LadvGG.The discriminator’s confrontation function is(3)LadvDG=−Ext∼pxt|ct,ct∼pctlogDxt,ct−Exs∼pxs|cs,ct∼pctlog1−DGxs,ct,ct.Among them,xt∼pxt|ct represents the spectrum characteristics of the target speaker, Dxt,ct represents the discriminator D to determine whether the real spectrum characteristics are true, Gxs,ct represents the spectrum characteristics of the target speaker generated by the generator G, and DGxs,ct,ct represents whether the spectrum characteristics generated by the discriminator are true, Ext∼pxt|ct,ct∼pct⋅ represents the expectation of conforming to the probability distribution of the true feature, and Exs∼pxs|cs,ct∼pct⋅ represents the expectation of conforming to the probability distribution of the generated feature.The confrontation loss function of generator is(4)LadvGG=−Exs∼pxs,ct∼pctlogDGxs,ct,ct.Among them,Gxs,ct represents the spectrum characteristics generated by the generator, DGxs,ct,ct represents whether the generated spectrum characteristics are true by the discriminator, and Exs∼pxs,ct∼pct⋅ represents the expectation of the probability distribution of the generated feature.The classifier is used to determine which speaker the generated speech belongs to. First, the classifier is trained with real speaker’s speech features, and the classification lossLclsCC of classifier is calculated. Furthermore, the classifier is used to judge the generated speaker’s speech category, calculate the classification loss LclsGG of the generator, and train the generator.The classification loss of the classifier is(5)LclsCC=−Ext∼pxt|ct,ct∼pctlogPCct|xs.The classification loss of the generator is(6)LclsGG=−Exs∼pxs,ct∼pctlogPCct|Gxs,ct.Among them,PCct|xs represents the probability that the classifier distinguishes the target speaker’s spectrum characteristics label as the target speaker’s label ct. PCct|Gxs,ct represents the probability that the classifier discriminates that the target speaker’s spectrum characteristics label is the target speaker label ct, and Gxs,ct represents the target speaker’s spectrum characteristics generated by the generator.The generator’s cycle consistency loss and feature mapping loss, the generator’s classification loss, and the generator’s confrontation loss are integrated. The total loss function of the generator is(7)IGG=LadvGG+λclsLclsGG+λcycLcycG+λidLidG.Among them,λcls≥0,λcyc≥0,λid≥0 is a regularization parameter, which, respectively, represents the weights of classification loss, cycle consistency loss, and feature mapping loss, and LadvGG,LclsGG,LcycG,LidG represent the generator’s confrontation loss, generator’s classification loss, cycle consistency loss, and feature mapping loss, respectively.The Star GAN-based speech conversion system includes two stages of training and conversion, as shown in Figure2.Figure 2
The training and conversion flowchart of Star GAN.In the training phase, through the WORLD speech analysis/synthesis model, the spectrum characteristicsx, fundamental frequency features logf0, and aperiodic features of each speaker’s sentence are extracted. Then, the spectrum characteristics xs of the source speaker, the spectrum characteristics xt of the target speaker, the source speaker label cs, and the target speaker label ct are input into the Star GAN for training. In the training process, it is necessary to make the loss function of the generator, the loss function of the discriminator, and the loss function of the classifier as small as possible until the set number of iterations, so as to obtain a trained Star GAN. The fundamental frequency conversion function from the fundamental frequency of the source speaker’s speech to the fundamental frequency logf0t of the target speaker’s speech is constructed.In the conversion stage, the source speaker’s speech in the corpus to be converted is used to obtain the spectral envelope characteristicsxs′, aperiodic features, and fundamental frequency logf0s′ through the WORLD speech analysis/synthesis model. Then, the source speaker’s spectral envelope characteristics xs′ and target speaker’s label characteristics ct′ are input into the trained Star GAN to reconstruct the target speaker’s spectral envelope characteristics xtc′. Then the fundamental frequency of the source speaker’s speech logf0s′ is converted into the fundamental frequency logf0t′ of the target speaker through the trained fundamental frequency conversion function. Finally, the target speaker’s spectral envelope characteristics xtc′, the fundamental frequency, and aperiodic characteristics of the target speaker are synthesised through the WORLD speech analysis/synthesis model to obtain the converted speaker’s speech.The model consists of three parts: a generatorG that generates the spectrum of the target speaker, a discriminator D that determines whether the input is a real spectrum or a generated spectrum, and a classifier C that determines whether the label of the spectrum belongs to the speaker. Among them, the generator G is composed of a precoding network, a multiscale module, and a decoding network. The network structure diagram of multiscale Star GAN is shown in Figure 3.Figure 3
Network structure diagram of multiscale Star GAN.The speech featurexs and target speaker label ct are input into the precoding network of the generator G containing the multiscale module to obtain the global characteristics of the target speaker domain. After that, the global features are imported into the multiscale module. Through the multiscale module, the range of the network’s receptive field is increased, and features of different levels are more fully extracted. In the multiscale module, the global feature Gxt is divided into s feature map subsets. Each feature map subset represents a scale and level of features, from the perception information of feature map subsets of different scales. Moreover, residual connections are used in the input and output of multiscale to build a hierarchical connection realization and alleviate the problem of gradient disappearance. The s feature map subsets are spliced to obtain hierarchically connected multiscale features GMxt, which are input to the decoder of the generator to obtain the spectrum characteristics xtc of the target speaker.The spectrum characteristicsxtc of the target speaker and the label characteristics cs of the source speaker are reinput to the precoding network of the generator G containing the multiscale module to obtain the global characteristics Gxs of the source speaker domain. After that, the hierarchical connected multiscale feature GMxs is obtained through multiscale coding. It is input to the decoder of the generator to obtain the spectrum characteristics xsc of the reconstructed source speaker, thereby obtaining the cycle loss of the spectrum characteristics xs of the source speaker and the spectrum characteristics xsc of the reconstructed source speaker.The cyclic consistent loss function is(8)LcycG=Exs∼pxs|cs,cs∼pcs,ct∼pctGGxs,ct,cs−xs1.Among them,xs∼pxs|cs represents the spectrum characteristics of the source speaker, cs∼pcs represents the label of the source speaker, and ct∼pct represents the label of the target speaker, GGxs,ct,cs represents the frequency spectrum characteristics of the reconstructed source speaker, and Exs∼pxs|cs,cs∼pcs,ct∼pct⋅ is the expected loss of the reconstructed source speaker’s spectrum and the real source speaker’s spectrum.In addition, the source speaker’s spectrum characteristicsxs and source speaker’s label characteristics cs are input to the generator to obtain the source speaker’s spectrum characteristics xss after feature mapping, so as to obtain the generator feature mapping loss Lid. The feature mapping loss function is(9)LidG=Ecs∼pcs,xs∼pxs|csGxs,cs−xs1.Among them,Gxs,cs is the source speaker spectrum characteristics obtained after the source speaker’s spectrum and source speaker’s label are input to the generator, and Ecs∼pcs,xs∼pxs|cs⋅ is the expected loss of xs and Gxs,cs.The discriminator is used to determine if the actual/generated feature conforms to the feature distribution of the real speaker’s speech, and it outputs the probability that the speech feature conforms to the real speaker’s speech feature distribution. The enhanced discriminator just accepts real/created speech and no longer requires the speech speaker’s tag; thus the source speaker’s actual speech is used as the real speech input, and the produced target speaker’s speech is used as the false speech input discriminator. The pretraining discriminator distinguishes real speech features and generated speech features and minimizes the discriminator’s confrontation loss functionLadvDG. The posttraining discriminator enables the generated speech features to deceive the discriminator into real speech and minimizes the discriminator’s confrontation loss function LadvGG.The confrontation loss function of the discriminator is(10)LadvDD=−Exs∼pxs|cs,cs∼pcslogDxs−Exs∼pxs,ct∼pctlog1−DGxs,ct.


## 4. Music Teaching System Based on Speech Recognition and Artificial Intelligence

On the basis of the above analysis, a music teaching system based on speech recognition and artificial intelligence is constructed. The system frame structure is shown in Figure4.Figure 4
System structure diagram.The basic music theory knowledge display submodule, the music basic theory knowledge retrieval submodule, and the music basic theory knowledge query submodule are all part of the basic music theory knowledge learning module. The display submodule is the system’s principal module for supporting basic music theory learning, and it may show knowledge items and specialised material related to basic music theory. The retrieval and query submodule allows you to search for and retrieve relevant information. Figure5 depicts the music teaching system’s system function flowchart, which is based on voice recognition and artificial intelligence.Figure 5
System function flowchart.The detailed design of the user management module is shown in Figure6:Figure 6
Function flowchart of the user management module.Within the user administration module, the logical link is rather straightforward. After passing the login verification, the system administrator may go straight to the system administration page and utilise the features of adding, changing, and removing users. If the login verification fails, the system switches to a reverification interface. The system administrator category, user management form category, system administrator information category, teacher user information category, student user information category, and process control category are the key categories defined by the module.


## 5. Performance Detection of the Music Teaching System Based on Speech Recognition and Artificial Intelligence

After constructing a music teaching system based on speech recognition and artificial intelligence technology, the performance of the system constructed in this paper is tested. According to the actual needs and requirements of the construction of the system in this paper, the performance of the system is tested from two aspects: speech recognition and system teaching effect.This paper uses 60 students from a university as a research sample to design experiments to analyze the performance of music teaching system. First, this paper analyzes the speech recognition effect of the system constructed in this paper, and the results are shown in Table1 and Figure 7.Table 1
Statistical table of speech recognition effect of the music teaching system.
NumberSpeech recognitionFeature extractionNumberSpeech recognitionFeature extraction192.466.53190.067.8283.767.03283.767.3392.666.43390.266.6487.666.43489.662.9588.769.93587.067.0688.365.23685.362.8788.161.33791.567.6883.462.63883.569.4992.369.23981.862.71084.563.04086.767.51186.069.74182.264.31292.563.94286.564.21380.764.94386.362.41480.262.94483.567.91582.065.94582.168.41682.363.64690.268.71781.563.34791.065.81883.762.84884.169.71981.963.14982.363.52080.762.85084.668.42183.569.15181.064.72284.963.05290.964.82389.261.75388.763.62482.066.55492.561.32580.364.75586.169.52685.067.35687.567.12791.661.05786.765.42891.664.65880.861.42982.561.15979.161.73091.368.56090.767.4Figure 7
Statistical diagram of speech recognition effect of the music teaching system.The above results show that the speech recognition ability of the music teaching system constructed in this paper is better. On this basis, this paper uses the teaching scoring method to verify the teaching performance of the music teaching system constructed in this paper. The results obtained are shown in Table2 and Figure 8.Table 2
Statistics table of music teaching effect evaluation of the music teaching system.
NumberTeaching effectNumberTeaching effectNumberTeaching effect172.32178.84188.1282.22287.64284.0385.72374.04377.3478.42471.44471.9574.52570.54574.1665.22687.14680.2772.12768.24780.5884.12889.14880.5971.82977.44972.11070.73075.35072.41182.83182.15173.71278.73271.15265.41381.33367.95377.91487.73486.35485.81589.33587.65585.61674.23688.65682.81785.03768.45771.21875.23886.75879.81968.33980.05966.12065.74078.16066.5Figure 8
Statistics diagram of music teaching effect evaluation of the music teaching system.Through the above experimental results, it can be seen that the music teaching system based on speech recognition and artificial intelligence constructed in this paper has greater advantages than the traditional teaching mode.


## 6. Conclusion

The maturation of computer network technology, as well as the steady use of campus networks, has laid a solid platform for the informatization of music education and instruction. Computer and network technologies may help with music education and teaching in ways that conventional classroom education and teaching cannot. As a result, an online music education and teaching assistant system that matches the requirements of modern educational technology development is required. In terms of the relevance of instructional system design theory, the essential position of instructional system design theory in the instructional design discipline is explained via in-depth discussion. Furthermore, the parallels and contrasts between teaching system design theory and teaching theory are examined in this work. Furthermore, this research designs a music education system based on voice recognition and artificial intelligence to validate the system’s performance using speech recognition technology. The results of the tests suggest that the approach proposed in this research is successful in music instruction.



---

*Source: 3900336-2022-03-18.xml*