# Innovative Construction of Reinforcement Learning Model for Information Fusion in Music Education

Authors: Zehui Zhang
Journal: Security and Communication Networks
 (2022)

---

## Abstract

In order to improve the effect of modern music education, this paper applies the digital information technology of music resources to the construction of the music teaching system and derives two new types of semi-decision-making process reinforcement learning algorithms based on the Bellman optimality equation base on discrete time. Moreover, this paper uses the comparative research methods to obtain the Q-value learning curve of the incremental value iterative reinforcement learning algorithm based on the semi-Markov decision process and the incremental value iterative reinforcement learning algorithm based on the dichotomy to improve the fusion effect of music teaching resources. Finally, this paper combines the actual needs of modern music education to construct an intelligent music teaching model.


---

## Body

## 1. Introduction

Music language plays an important role in the music development of a country. The mother tongue is one or several languages that a person first contacts, learns, and masters [1]. The mother tongue is generally contacted from a young age and continues to be used in adolescents or later. Moreover, in a person’s family or formal education, especially in the early stages, a considerable part of the knowledge is imparted through the mother tongue. The language of our country is mainly Chinese, and the music form is mainly Chinese folk songs and the five-tone mode of musical instruments [2]. However, the current universal music language in the world is mainly based on the Western education system, which puts a test for us to understand the world’s music culture and creates difficulties for us to convey, learn, and understand the world’s excellent music culture. Therefore, modern national educational institutions propose that music culture should inherit and carry forward China’s excellent traditional culture, strive to learn the Western music culture knowledge, and integrate the advantages of the two aspects to establish a world cultural form with Chinese cultural heritage. This also puts forward higher requirements for our country’s music education in the world and also establishes the world status that our country’s music education must achieve [3].From a broad perspective, the ongoing music education in China is largely based on the level of teachers teaching students to learn, and teachers only understand the teaching of students simply listening to music. This traditional teaching mode has been going through for a long time. This kind of teaching method dominated by the transmission of knowledge is often easy to teach knowledge to death, and it is difficult for students to cultivate character and develop personality. Although I have learned a lot of music knowledge, it is easy to focus on music knowledge and ignore the deeper aspects of music.At present, the overall development of China’s music education industry is facing great challenges, and the trend of international music education cannot be responded quickly; knowledge updating lags behind; music education has not been solved at a higher level; the publication of academic works is extremely rare; China rarely participates in international music education cooperation, and the number of participants is also small. In addition, local traditional music teaching and teacher education in the world, such as oriental music teaching, are considered to be vacant in many fields. It is undeniable that the cultural construction pattern of global integration is inevitable. It has made a positive response to the development of human science and technology, culture, and economy. The living space has been gathered and developed. Therefore, it is not profound for any modern country to abandon its own development or abandon its existing cultural traditions to talk about the development of the world pattern, and it will face many difficulties in the future. Chinese music education first needs to consider the actual background of China’s digital music teaching and the development trend of current international music education, such as the mother tongue problem, the problem of education teachers, and the update of knowledge, which have very important historical significance for modern education in my country.Due to the rapid development of multimedia technology, a large amount of information disseminated on the Internet is in different modalities. For example, for any web page, whether it is about sports, Weibo, or military politics, it will basically contain audio, video, and images, and any one. All modal data information is indispensable. The data information of these different modalities are all developed around the theme of the web page, so cross-modal analysis can be used, that is, the data of different modalities can be found in a certain way to find the correlation between them, so as to carry out comparative analysis. For example, when people listen to a piece of music, the music contains text lyrics that describe the content of the music. These audios and their corresponding text lyrics can be regarded as bimodal information, and they are related to each other in emotional expression. Especially in the era of big data and cloud computing sweeping the network and research in various fields, cross-modal analysis has become a hot spot in this field and has received more and more attention. Therefore, we need to continue to explore and analyze it.The research and application of traditional single-modal data has been quite extensive, but in the era of multimedia data and big data becoming the mainstream, users’ requirements for data are getting higher and higher, and the analysis of single-modal data can no longer keep up with the needs of the times, and people can no longer conduct broader research based on it. Because of the limitation of single modality data, the data information contained in it cannot be fully utilized by people and it cannot describe multimedia data more comprehensively. For multimodal data, they are often complementary to each other, and can express better results when characterizing objects.The organization structure of this paper is as follows: The first part introduces the related concepts and background of music education and multimodal fusion analysis. The second part is the literature review part, which describes the main work of the music multimodal fusion analysis at home and abroad at present and the music multi-modal fusion analysis. The current situation and difficulties are introduced. The third part proposes an incremental value iterative reinforcement learning algorithm based on the needs of music education, and the algorithm is used to construct a multimodal fusion analysis model for music. The fourth part is based on the support of the third part. The multimodal information fusion music education system is used for learning. The fifth part is to test the validity of the model in this paper by means of experiments. The conclusion part is a summary of the research results of this paper and puts forward the prospect.The main contributions of this paper are as follows: (1) according to the iterative form of the Bellman optimality equation, a unified analysis framework for SMDP reinforcement learning algorithm is given, which can effectively promote the fusion of multimodal music information; (2) reinforcement learning under the average reward criterion algorithms to carry out related research, so that the music information fusion reinforcement learning algorithm can be more widely used in practical systems.This paper uses the reinforcement learning model based on information fusion to innovate the way of music education, change the traditional music education model, and improve the actual effect of music education.


## 2. Related Work

Due to the rapid development of multimedia technology, a large amount of information disseminated on the Internet is of different modalities. For example, for any webpage, whether it is about sports, Weibo, or military politics, it basically contains audio, video, and images, and any one of them. The data information of each modal is indispensable. Moreover, the data information of these different modalities is developed around the theme of the web page, so cross-modal analysis can be used; that is, the data of different modalities can be found in a certain way to find the relationship between them, so as to conduct a comparative analysis. For example, when people listen to a piece of music, the music contains text lyrics that describe the music content. These audio and the corresponding text lyrics can be regarded as bimodal information, and they are related to each other in emotional expression [4]. Especially in the era when big data and cloud computing are sweeping the network and research in various fields, cross-modal analysis has become a hot spot in this field and has attracted more and more attention. Therefore, we need to continue to explore and analyze it. For cross-modal analysis, Cano [5] found the correlation between the two-modal data of text and image through the joint learning of the two-modal data and improved the accuracy of image search, text search, and multi-modal data search. In order to eliminate the duality of semantic expression of text information and improve the accuracy of expression, the text information is correlated with visual characteristics, and the text is expressed more semantically [6]. Dickens [7], based on the Probabilistic Semantic Model (PLSA), learns and trains multimodal information, obtains a hierarchical representation mechanism of multiple modal data, finds the correlation information of data between multiple modalities, and improves data search and query efficiency. Gonçalves [8] based on the multimodal LPP algorithm, trains the music and image sample sets to obtain a common low-dimensional subspace, in which the same sample and different modal data will be close together, that is, multiple modal data. Relevance expression can be carried out, which eliminates the gap in semantic relevance between multiple modal data.The research and application of traditional single-modal data has been quite extensive, but in the era when multimedia data and big data have become the mainstream, users have higher and higher requirements for data, and the analysis of single-modal data can no longer keep up with the needs of the development of the times. People can no longer carry out a wider range of research based on it. Because of the limitations of single-modal data, the data information it contains cannot be fully utilized by people, and it cannot describe multimedia data more comprehensively. For multimodal data, they are often complementary and interrelated and can express better results when portraying objects [9]. In the research on the automatic generation of multimodal fusion family music albums, by fusing two different modal data of image and music, it played a key role in the semantic understanding of music and images, which has a significant impact on the research results [10]. For multi-modal fusion, Gorbunova [11] compares music and images through the analysis and training of graphs and obtains good results in practical applications. Khulusi [12] based on the principle of Bernoulli distribution aimed to annotate images and texts. Magnusson [13] based on deep learning RBM (Restricted Boltzmann Machines) and DBN (Deep Belief Network) aimed to provide a common feature representation for multimodal data and apply it in various recognition fields. Partesotti [14] used the mutual mapping between text and image to find the mutual relationship between them, and then carried out text identification on the image. Scavone [15] proposed a supervised learning model, which is an improvement on the unsupervised model. It maps multimodal data to a subspace, in which the same type of data will moved closer to each other. In addition, multimodal fusion has also been widely used in classification tasks and the purpose is to improve the accuracy of classification. Serra [16] classifies text, audio, and video features through a fusion model and has achieved good results in the experimental results. Tabuena [17] based on the mapping technology spatially mapped and classified multimodal data, which effectively proves the accuracy of multimodal fusion. Tomašević [18] uses the BOW model to generate file-level lyrics, which are then fused with the extracted audio and video features, and performs semantic classification. In short, multimodal fusion will play an increasingly important role in future research.The working principle of the emotion model [19] is as follows: first extract emotion words from the emotion library, mainly including some words and sets, and then classify these words and describe the emotions hidden in music according to the classification results. Although this model can better express the semantic information of music, it is not suitable for the extraction of music audio features. Therefore, this model is not suitable for the study of music emotion judgment in this article.


## 3. Incremental Value Iterative Reinforcement Learning Algorithm Based on Music Education Needs

### 3.1. Markov Decision Process Reinforcement Learning

Markov decision process reinforcement learning can improve the effect of music information decision-making, so this paper first analyzes the Markov decision process reinforcement learning.In the discrete-time Bellman optimality equation, since the optimal average returnηu∗ is unknown, the value iteration algorithm cannot be directly obtained by the formula. Based on the above analysis, it can be seen that the optimal average return ηu∗ needs to be estimated, and the incremental value iteration algorithm belongs to the direct estimation method. The core idea of the direct estimation method is to directly estimate the optimal average return ηu∗ in the process of value iteration.We useη˜ to represent the estimated value of the optimal average return ηu∗, and Bu and Bl represent the upper and lower bounds of the stay τui. Then, for any strategy μ∈∏s [20],(1)1≤Bl≤nμ≤Bu.In the abovementioned formula,(2)nμ=πμτμ.It can be seen from the formula that the average returnημ under the semi-Markov decision process can be regarded as the ratio of the average return πμrμ and πμτμ. The performance function of the embedded chain is set as follows:(3)fμ=1nμrμ.Combining the abovementioned formula, we can getπμfμ=ημ, so the embedded chain and the original Markov chain have the same average reward. This can be seen as the entire segment is concentrated on the embedded decision point, so the essence of the time-concentrated Markov decision process is to introduce a reward function into the embedded chain. Since the performance function f is not only related to the state but also related to the actions taken in the state, the performance function f cannot be solved directly. In order to be able to perform the iterative process of updating the strategy on the embedded chain, this topic defines the following performance function [21]:(4)fi,a=ri,a−η˜⋅τi,a.The performance potentialgμ can be calculated by using the following formula:(5)gμ=maxμ∈∏srμ−η˜τμ+Pμgμ.Subsequently, the optimal strategy u∗ can be obtained using the following strategy update process:(6)μ∗=argmaxμ∈∏srμ−η˜τμ+Pμgμ.Through the repeated iteration of formulas (5) and (6), the abovementioned strategy iteration algorithm can be obtained by the optimal strategy under the original semi-Markov decision process. In the iterative process, as the strategy μ approaches the optimal strategy μ∗, the estimated value also continues to approach the optimal average return ημ∗. Therefore, the performance function f(i,a) will also keep changing during the process of strategy iteration.The optimal strategyϕ∗ of the abovementioned time-focused Markov decision process can be obtained by the following formula [22]:(7)ϕ∗=argmaxμ∈∏sΔημ.Among them, there are as follows:(8)Δημ=πμ⋅rμ−η˜⋅τμ.Δημ can be rewritten as follows:(9)Δημ=ημ−η˜⋅nμ.It can be seen from formula (7) that the optimal strategy p is related to the estimated value i of the optimal average return. The optimal strategy is analyzed below.Since the strategy ui is the optimal strategy under the original semi-Markov decision process,ημ∗≥ηϕ∗. According to formula (9), we can obtain the following:(10)Δηϕ∗=ηϕ∗−η˜≤ημ∗−η˜⋅ηϕ∗.In addition, because the strategyϕ∗ is the optimal strategy of the Markov decision process in time concentration, Δηϕ∗≥Δημ∗. Using formula (9), we can obtain the following:(11)Δηϕ∗≥ημ∗−η˜⋅ημ∗.Combining formulas (10) and (11), the following relationship can be obtained:(12)Δηϕ∗nϕ∗≤ημ∗−η˜≤Δηϕ∗nμ∗.Ifη˜≥ημ∗, the following inequality can be obtained by formula (10):(13)Δηϕ∗=ημ∗−η˜⋅nϕ∗≤0.According to the abovementioned formula, the following relational formula can be obtained:(14)Δηϕ∗≤η˜.Ifη˜≤ημ∗, combining formulas (9) and (11), we can obtain the following:(15)Δηϕ∗≥ημ∗≥0.By observing (13) and (15), the following conclusions can be drawn: when η˜≥ημ∗, Δηϕ∗≤0 is established, when η˜≤ημ∗, Δηϕ∗≥0 is established. In summary, Δηϕ∗ can provide the correct search direction for the optimal average return. Subsequently, the estimated value ημ∗ of the optimal average return η˜ can be updated iteratively through one-dimensional search, and the process is as follows:(16)η˜←η˜+H⋅Δηϕ∗,H=1Bμ.In the abovementioned formula, H is the search step size of the estimated valueη˜.


### 3.2. SMDP Incremental Iterative Reinforcement Learning Algorithm

The extended form of the SMDP incremental value iterative reinforcement learning algorithm is the same as the R-learning algorithm, which is derived from the discrete-time Bellman optimality equation. Therefore, the SMDP incremental value iterative reinforcement learning algorithm’s state-action-value function update formula is the same as that of the R-learning algorithm, and the specific process is shown in Algorithm 1.(1)
InitializeQ (state action to Q-value) expressed as any real number, usually make Q0(i, a) = 0, ∀i,a∈S×A.The algorithm setstm = tn = 0 and specifies ε>σ>0 and initializes η˜0 as any real number.(2)
The algorithm calculatesQtm+1Xn,An for each state Xn∈S:(17)Qtm+1Xn,An←1−α⋅QtmXn,An+α⋅RXn,An−η˜⋅τXn,An+maxa′∈AQtmXn+1,a′.(3)
IfspQtm+1−Qtm≤σ, the algorithm executes step 4. sp(Q) is the span of Q, as shown below:(18)spQ≡maxiÎSmaxaÍAiQi,a−miniÎSmaxaÍAiQi,a.Otherwise, the algorithm setstm←tm+1, n←n+1 and executes Step 2.(4)
The algorithm calculatesΔηϕ∗:(19)12⋅maxi∈Smaxa∈AiQtm+1i,a−maxa∈AiQtmi,a+mini∈Smaxa∈AiQtm+1i,a−maxa∈AiQtmi,a.IfΔηϕ∗≤ε, the algorithm executes Step 5. Otherwise, the algorithm updates η˜.(20)η˜tm+1←η˜tmH⋅Δηϕ∗,H=1Bμ.The algorithm sets uptn←tn+1, tm←tm+1, n←n+1 and returns to Step 2.(5)
For any statei s ∈ S, the algorithm selects the optimal strategy μ∗ through the following formula:(21)μ∗i∈argmaxa∈AiQtmi,a.Inspired by formula (16), since Δηϕ∗ can provide the correct search direction for the optimal average return, we can use the dichotomy to replace formula (16) to estimate the optimal average return directly. The algorithm gives two initial estimates element and element, so that it satisfies the relationship η˜1≤ημ∗≤η˜2. The algorithm uses η˜≤ημ∗≤η˜2=η˜1+η˜2 as the initial estimate of the optimal average return Δημ∗. If Δηϕ∗>0, the algorithm sets η˜1←η˜; otherwise, the algorithm sets η˜2←η˜. Using the abovementioned coin update method instead of formula (20) in algorithm 1, an iterative reinforcement learning algorithm based on the dichotomy of incremental values can be obtained. Due to the introduction of the dichotomy, the IVI reinforcement learning algorithm based on the dichotomy also has a higher optimization efficiency in the case of a large state action space. Algorithm flowchart is shown in Figure 1.Figure 1
Algorithm flowchart.


### 3.3. Reinforcement Learning Curve Analysis

This section gives the Q-value learning curve of the incremental value iterative reinforcement learning algorithm for the semi-Markov decision process and the incremental value iterative reinforcement learning algorithm based on the dichotomy. The simulation experiment results of these two algorithms are shown in Figures1 and 2, respectively. The learning rate α of the incremental value iterative reinforcement learning algorithm and the incremental value iterative reinforcement learning algorithm based on dichotomy are both 1/kXn,An+1, and the parameters s and α are 10∗ and 10, respectively. The search step size of the incremental value iterative reinforcement learning algorithm is H = 0.1, and the initial optimal average return estimate η˜0=4. The initial estimates of the incremental value iterative reinforcement learning algorithm based on dichotomy η˜1 and η˜2 are η˜1=0 and η˜2=6, respectively, and the initial estimated value of the optimal average return ημ∗ is η˜=3.Figure 2
Learning curve. (a) Q-value learning curve of IVI reinforcement learning algorithm. (b) Q-value learning curve of IVI reinforcement learning algorithm based on dichotomy.
(a)(b)As can be seen from Figures2(a) and 2(b), the IVI reinforcement learning algorithm and the dichotomy-based IVI reinforcement learning algorithm both converge within 40,000 iteration steps. Among them, the IVI reinforcement learning algorithm converges faster than the IVI reinforcement learning algorithm based on the dichotomy.In this paper, the incremental value iteration algorithm is used to directly estimate the optimal average return. In addition, this section uses the SSP value iterative algorithm to directly estimate the optimal average reward, so as to obtain the SMDP random shortest path value iterative reinforcement learning algorithm. The specific process of the random shortest path problem is shown in Figure3.Figure 3
Example of a random shortest path problem.As can be seen from the abovementioned figure, the random shortest path problem sets the transition probabilitypμii,ξ to a special state ξ to zero, while the transition probability pμii,ξ in other states remains unchanged. Artificially introduce the absorption termination state t, and set the transition probability of any state i to the termination state t as pμii,ξ. The expected reward function rSSPμi of the random shortest path problem under the state i∈S is defined as follows:(22)rSSPμi=rμi−η˜⋅τμi.The algorithm assumes thatgμi represents the expected total reward of the random shortest path problem under strategy u starting from state i. Subsequently, there is the following relationship:(23)gμi−∑j=1ξ−1pμi,j⋅gμi=rμi,μi−η˜⋅τμi,μi.Both sides of the abovementioned formula are multiplied byπμi to the left at the same time, and then they are accumulated and summed from i = 1 to ξ to obtain the following formula:(24)πμi⋅gμi=πμi⋅rμi,μi−η˜⋅τμi,μi=Δημ.From the abovementioned formula, the following relationship can be derived:(25)gμξ=Δημπμξ.The optimal strategyφ∗ for the random shortest path problem can be obtained by the following formula:(26)φ∗=argmaxμ∈∏sgμξ.The derivation process of the iterative update formula of the SSP value iterative algorithm coin is similar to the algorithm. Combining (2) and (9), we can see the following:(27)Δημ=πμτμ⋅ημ−η˜.According to the optimality of the strategy u∗, we know that ημ∗≥ηφ∗. Combining formulas (25) and (27), we can obtain the following:(28)gφ∗ξ=Δηφ∗πφ∗ξ=πφ∗τφ∗πφ∗ξ⋅ηφ∗−η˜≤πφ∗τφ∗πφ∗ξ⋅ηφ∗−η˜.According to the optimality of strategyφ∗, we know that Δηφ∗≥Δημ∗. Similar to the above analysis, we can obtain the following:(29)gφ∗ξ=Δηφ∗πφ∗ξ≥Δημ∗πφ∗ξ=πφ∗τφ∗πφ∗ξ⋅ημ∗−η˜.Under the assumption of ergodicity, according to (28) and (29), it can be known that when η˜≥ημ∗, gφ∗ξ≤0 holds, and when η˜≤ημ∗, gφ∗ξ≥0 holds. In summary, gφ∗ξ can provide the correct search direction for the optimal average return ημ∗. Therefore, the estimated value η˜ of ημ∗ can be updated iteratively by the following formula:(30)η˜←η˜+γ⋅gφ∗ξ,γ=1Bμ.In the abovementioned formula,γ is the search step length of the estimated value η˜. Subsequently, this subject obtained the random shortest path value iteration reinforcement learning algorithm under the semi-Markov decision process.(1)
The algorithm initializes theQ table (state action versus Q-value) to any real number, and the algorithm usually sets Q0(i,a) = 0, ∀i,a∈S×A.The algorithm setstm = tn = 0 and specifies ε>σ>0 and initializes η˜0 as any real number.(2)
For each stateXn∈S, the algorithm calculates Qtm+1Xn,An:(31)Qtm+1Xn,An←1−α⋅QtmXn,An+α⋅RXn,An−η˜⋅τXn,An+maxa′∈AQtmXn+1,a′⋅IXn+1≠ξ.Among them,I… is the indicator function.(3)
IfspQtm+1−Qtm≤σ, the algorithm executes step 4. Otherwise, the algorithm sets tm←tm+1, n←n+1 and executes step 2.(4)
The algorithm usesmaxa∈AiQtm+1ξ,a to calculate gφ∗ξ. If maxa∈AiQtm+1ξ,a≤ε, the algorithm executes step 5. Otherwise, the algorithm updates η˜.(32)η˜tn+1←η˜tn+γ⋅maxa∈AQtm+1,γ=1Bμ.We settn←tn+1, tm←tm+1, n←n+1 and return to Step 2.(5)
For any statei ∈ S, the algorithm selects the optimal strategy μ∗ by the following formula.(33)μ∗i∈argmaxa∈AiQtmi,a.Sincegφ∗ξ can provide the correct search direction for the optimal average reward ημ∗, the dichotomy can also be used to replace formula (30) to directly estimate the optimal average reward ημ∗. The algorithm gives initial estimates η˜1 and η˜2, if gφ∗ξ>0, the algorithm sets η˜1←η˜, otherwise, the algorithm sets η˜2←η˜. Using the abovementioned update method instead of formula (32) in algorithm 2, an iterative reinforcement learning algorithm based on the dichotomy of SSP values can be obtained.The Q-value learning curve of the SMDP random shortest path value iterative reinforcement learning algorithm and the random shortest path value iterative reinforcement learning algorithm based on the dichotomy are given, respectively. The simulation experiment results of the algorithm are shown in Figures4 and 5, respectively. The simulation environment of these two algorithms is the same as that of the IVI reinforcement learning algorithm. The learning rate of SSP value iterative reinforcement learning algorithm α, parameters s and o, and the estimated value of the initial optimal average return are the same as the IVI reinforcement learning algorithm, and its search step γ=0.1. The initial estimates η˜1 and η˜2 of the SSP value iterative reinforcement learning algorithm based on the dichotomy are the same as those of the IVI reinforcement learning algorithm based on the dichotomy. The special state in the random shortest path problem is ξ=1.Figure 4
Q-value learning curve of SSP value iterative reinforcement learning algorithm.Figure 5
Q-value learning curve of SSP value iterative reinforcement learning algorithm based on dichotomy.It can be seen from Figures4 and 5 that the SSP value iterative reinforcement learning algorithm and the dichotomy-based SSP value iterative reinforcement learning algorithm both converged within 22,000 iteration steps. Among them, the SSP value iterative reinforcement learning algorithm converges faster than the SSP value iterative reinforcement learning algorithm based on the dichotomy. The SSP value iterative reinforcement learning algorithm has the fastest convergence speed and the best convergence performance. Compared with other semi-Markov reinforcement learning algorithms in this article, the SSP value iterative reinforcement learning algorithm has the highest efficiency in finding the optimal strategy. By comparing Figures 1 and 4, it can be seen that the SSP value iterative reinforcement learning algorithm has improved the convergence performance of the IVI reinforcement learning algorithm. The simulation results in Figures 4 and 5 verify the convergence of the SSP value iterative reinforcement learning algorithm and the SSP value iterative reinforcement learning algorithm based on the dichotomy. Although the effect of using the dichotomy to directly estimate the optimal average return ημ∗ is not as good as the SSP value iterative reinforcement learning algorithm. However, the iterative reinforcement learning algorithm of SSP value based on dichotomy compared with the semi-Markov reinforcement learning algorithm in this paper still has a large degree of improvement in convergence performance.


## 3.1. Markov Decision Process Reinforcement Learning

Markov decision process reinforcement learning can improve the effect of music information decision-making, so this paper first analyzes the Markov decision process reinforcement learning.In the discrete-time Bellman optimality equation, since the optimal average returnηu∗ is unknown, the value iteration algorithm cannot be directly obtained by the formula. Based on the above analysis, it can be seen that the optimal average return ηu∗ needs to be estimated, and the incremental value iteration algorithm belongs to the direct estimation method. The core idea of the direct estimation method is to directly estimate the optimal average return ηu∗ in the process of value iteration.We useη˜ to represent the estimated value of the optimal average return ηu∗, and Bu and Bl represent the upper and lower bounds of the stay τui. Then, for any strategy μ∈∏s [20],(1)1≤Bl≤nμ≤Bu.In the abovementioned formula,(2)nμ=πμτμ.It can be seen from the formula that the average returnημ under the semi-Markov decision process can be regarded as the ratio of the average return πμrμ and πμτμ. The performance function of the embedded chain is set as follows:(3)fμ=1nμrμ.Combining the abovementioned formula, we can getπμfμ=ημ, so the embedded chain and the original Markov chain have the same average reward. This can be seen as the entire segment is concentrated on the embedded decision point, so the essence of the time-concentrated Markov decision process is to introduce a reward function into the embedded chain. Since the performance function f is not only related to the state but also related to the actions taken in the state, the performance function f cannot be solved directly. In order to be able to perform the iterative process of updating the strategy on the embedded chain, this topic defines the following performance function [21]:(4)fi,a=ri,a−η˜⋅τi,a.The performance potentialgμ can be calculated by using the following formula:(5)gμ=maxμ∈∏srμ−η˜τμ+Pμgμ.Subsequently, the optimal strategy u∗ can be obtained using the following strategy update process:(6)μ∗=argmaxμ∈∏srμ−η˜τμ+Pμgμ.Through the repeated iteration of formulas (5) and (6), the abovementioned strategy iteration algorithm can be obtained by the optimal strategy under the original semi-Markov decision process. In the iterative process, as the strategy μ approaches the optimal strategy μ∗, the estimated value also continues to approach the optimal average return ημ∗. Therefore, the performance function f(i,a) will also keep changing during the process of strategy iteration.The optimal strategyϕ∗ of the abovementioned time-focused Markov decision process can be obtained by the following formula [22]:(7)ϕ∗=argmaxμ∈∏sΔημ.Among them, there are as follows:(8)Δημ=πμ⋅rμ−η˜⋅τμ.Δημ can be rewritten as follows:(9)Δημ=ημ−η˜⋅nμ.It can be seen from formula (7) that the optimal strategy p is related to the estimated value i of the optimal average return. The optimal strategy is analyzed below.Since the strategy ui is the optimal strategy under the original semi-Markov decision process,ημ∗≥ηϕ∗. According to formula (9), we can obtain the following:(10)Δηϕ∗=ηϕ∗−η˜≤ημ∗−η˜⋅ηϕ∗.In addition, because the strategyϕ∗ is the optimal strategy of the Markov decision process in time concentration, Δηϕ∗≥Δημ∗. Using formula (9), we can obtain the following:(11)Δηϕ∗≥ημ∗−η˜⋅ημ∗.Combining formulas (10) and (11), the following relationship can be obtained:(12)Δηϕ∗nϕ∗≤ημ∗−η˜≤Δηϕ∗nμ∗.Ifη˜≥ημ∗, the following inequality can be obtained by formula (10):(13)Δηϕ∗=ημ∗−η˜⋅nϕ∗≤0.According to the abovementioned formula, the following relational formula can be obtained:(14)Δηϕ∗≤η˜.Ifη˜≤ημ∗, combining formulas (9) and (11), we can obtain the following:(15)Δηϕ∗≥ημ∗≥0.By observing (13) and (15), the following conclusions can be drawn: when η˜≥ημ∗, Δηϕ∗≤0 is established, when η˜≤ημ∗, Δηϕ∗≥0 is established. In summary, Δηϕ∗ can provide the correct search direction for the optimal average return. Subsequently, the estimated value ημ∗ of the optimal average return η˜ can be updated iteratively through one-dimensional search, and the process is as follows:(16)η˜←η˜+H⋅Δηϕ∗,H=1Bμ.In the abovementioned formula, H is the search step size of the estimated valueη˜.


## 3.2. SMDP Incremental Iterative Reinforcement Learning Algorithm

The extended form of the SMDP incremental value iterative reinforcement learning algorithm is the same as the R-learning algorithm, which is derived from the discrete-time Bellman optimality equation. Therefore, the SMDP incremental value iterative reinforcement learning algorithm’s state-action-value function update formula is the same as that of the R-learning algorithm, and the specific process is shown in Algorithm 1.(1)
InitializeQ (state action to Q-value) expressed as any real number, usually make Q0(i, a) = 0, ∀i,a∈S×A.The algorithm setstm = tn = 0 and specifies ε>σ>0 and initializes η˜0 as any real number.(2)
The algorithm calculatesQtm+1Xn,An for each state Xn∈S:(17)Qtm+1Xn,An←1−α⋅QtmXn,An+α⋅RXn,An−η˜⋅τXn,An+maxa′∈AQtmXn+1,a′.(3)
IfspQtm+1−Qtm≤σ, the algorithm executes step 4. sp(Q) is the span of Q, as shown below:(18)spQ≡maxiÎSmaxaÍAiQi,a−miniÎSmaxaÍAiQi,a.Otherwise, the algorithm setstm←tm+1, n←n+1 and executes Step 2.(4)
The algorithm calculatesΔηϕ∗:(19)12⋅maxi∈Smaxa∈AiQtm+1i,a−maxa∈AiQtmi,a+mini∈Smaxa∈AiQtm+1i,a−maxa∈AiQtmi,a.IfΔηϕ∗≤ε, the algorithm executes Step 5. Otherwise, the algorithm updates η˜.(20)η˜tm+1←η˜tmH⋅Δηϕ∗,H=1Bμ.The algorithm sets uptn←tn+1, tm←tm+1, n←n+1 and returns to Step 2.(5)
For any statei s ∈ S, the algorithm selects the optimal strategy μ∗ through the following formula:(21)μ∗i∈argmaxa∈AiQtmi,a.Inspired by formula (16), since Δηϕ∗ can provide the correct search direction for the optimal average return, we can use the dichotomy to replace formula (16) to estimate the optimal average return directly. The algorithm gives two initial estimates element and element, so that it satisfies the relationship η˜1≤ημ∗≤η˜2. The algorithm uses η˜≤ημ∗≤η˜2=η˜1+η˜2 as the initial estimate of the optimal average return Δημ∗. If Δηϕ∗>0, the algorithm sets η˜1←η˜; otherwise, the algorithm sets η˜2←η˜. Using the abovementioned coin update method instead of formula (20) in algorithm 1, an iterative reinforcement learning algorithm based on the dichotomy of incremental values can be obtained. Due to the introduction of the dichotomy, the IVI reinforcement learning algorithm based on the dichotomy also has a higher optimization efficiency in the case of a large state action space. Algorithm flowchart is shown in Figure 1.Figure 1
Algorithm flowchart.


## 3.3. Reinforcement Learning Curve Analysis

This section gives the Q-value learning curve of the incremental value iterative reinforcement learning algorithm for the semi-Markov decision process and the incremental value iterative reinforcement learning algorithm based on the dichotomy. The simulation experiment results of these two algorithms are shown in Figures1 and 2, respectively. The learning rate α of the incremental value iterative reinforcement learning algorithm and the incremental value iterative reinforcement learning algorithm based on dichotomy are both 1/kXn,An+1, and the parameters s and α are 10∗ and 10, respectively. The search step size of the incremental value iterative reinforcement learning algorithm is H = 0.1, and the initial optimal average return estimate η˜0=4. The initial estimates of the incremental value iterative reinforcement learning algorithm based on dichotomy η˜1 and η˜2 are η˜1=0 and η˜2=6, respectively, and the initial estimated value of the optimal average return ημ∗ is η˜=3.Figure 2
Learning curve. (a) Q-value learning curve of IVI reinforcement learning algorithm. (b) Q-value learning curve of IVI reinforcement learning algorithm based on dichotomy.
(a)(b)As can be seen from Figures2(a) and 2(b), the IVI reinforcement learning algorithm and the dichotomy-based IVI reinforcement learning algorithm both converge within 40,000 iteration steps. Among them, the IVI reinforcement learning algorithm converges faster than the IVI reinforcement learning algorithm based on the dichotomy.In this paper, the incremental value iteration algorithm is used to directly estimate the optimal average return. In addition, this section uses the SSP value iterative algorithm to directly estimate the optimal average reward, so as to obtain the SMDP random shortest path value iterative reinforcement learning algorithm. The specific process of the random shortest path problem is shown in Figure3.Figure 3
Example of a random shortest path problem.As can be seen from the abovementioned figure, the random shortest path problem sets the transition probabilitypμii,ξ to a special state ξ to zero, while the transition probability pμii,ξ in other states remains unchanged. Artificially introduce the absorption termination state t, and set the transition probability of any state i to the termination state t as pμii,ξ. The expected reward function rSSPμi of the random shortest path problem under the state i∈S is defined as follows:(22)rSSPμi=rμi−η˜⋅τμi.The algorithm assumes thatgμi represents the expected total reward of the random shortest path problem under strategy u starting from state i. Subsequently, there is the following relationship:(23)gμi−∑j=1ξ−1pμi,j⋅gμi=rμi,μi−η˜⋅τμi,μi.Both sides of the abovementioned formula are multiplied byπμi to the left at the same time, and then they are accumulated and summed from i = 1 to ξ to obtain the following formula:(24)πμi⋅gμi=πμi⋅rμi,μi−η˜⋅τμi,μi=Δημ.From the abovementioned formula, the following relationship can be derived:(25)gμξ=Δημπμξ.The optimal strategyφ∗ for the random shortest path problem can be obtained by the following formula:(26)φ∗=argmaxμ∈∏sgμξ.The derivation process of the iterative update formula of the SSP value iterative algorithm coin is similar to the algorithm. Combining (2) and (9), we can see the following:(27)Δημ=πμτμ⋅ημ−η˜.According to the optimality of the strategy u∗, we know that ημ∗≥ηφ∗. Combining formulas (25) and (27), we can obtain the following:(28)gφ∗ξ=Δηφ∗πφ∗ξ=πφ∗τφ∗πφ∗ξ⋅ηφ∗−η˜≤πφ∗τφ∗πφ∗ξ⋅ηφ∗−η˜.According to the optimality of strategyφ∗, we know that Δηφ∗≥Δημ∗. Similar to the above analysis, we can obtain the following:(29)gφ∗ξ=Δηφ∗πφ∗ξ≥Δημ∗πφ∗ξ=πφ∗τφ∗πφ∗ξ⋅ημ∗−η˜.Under the assumption of ergodicity, according to (28) and (29), it can be known that when η˜≥ημ∗, gφ∗ξ≤0 holds, and when η˜≤ημ∗, gφ∗ξ≥0 holds. In summary, gφ∗ξ can provide the correct search direction for the optimal average return ημ∗. Therefore, the estimated value η˜ of ημ∗ can be updated iteratively by the following formula:(30)η˜←η˜+γ⋅gφ∗ξ,γ=1Bμ.In the abovementioned formula,γ is the search step length of the estimated value η˜. Subsequently, this subject obtained the random shortest path value iteration reinforcement learning algorithm under the semi-Markov decision process.(1)
The algorithm initializes theQ table (state action versus Q-value) to any real number, and the algorithm usually sets Q0(i,a) = 0, ∀i,a∈S×A.The algorithm setstm = tn = 0 and specifies ε>σ>0 and initializes η˜0 as any real number.(2)
For each stateXn∈S, the algorithm calculates Qtm+1Xn,An:(31)Qtm+1Xn,An←1−α⋅QtmXn,An+α⋅RXn,An−η˜⋅τXn,An+maxa′∈AQtmXn+1,a′⋅IXn+1≠ξ.Among them,I… is the indicator function.(3)
IfspQtm+1−Qtm≤σ, the algorithm executes step 4. Otherwise, the algorithm sets tm←tm+1, n←n+1 and executes step 2.(4)
The algorithm usesmaxa∈AiQtm+1ξ,a to calculate gφ∗ξ. If maxa∈AiQtm+1ξ,a≤ε, the algorithm executes step 5. Otherwise, the algorithm updates η˜.(32)η˜tn+1←η˜tn+γ⋅maxa∈AQtm+1,γ=1Bμ.We settn←tn+1, tm←tm+1, n←n+1 and return to Step 2.(5)
For any statei ∈ S, the algorithm selects the optimal strategy μ∗ by the following formula.(33)μ∗i∈argmaxa∈AiQtmi,a.Sincegφ∗ξ can provide the correct search direction for the optimal average reward ημ∗, the dichotomy can also be used to replace formula (30) to directly estimate the optimal average reward ημ∗. The algorithm gives initial estimates η˜1 and η˜2, if gφ∗ξ>0, the algorithm sets η˜1←η˜, otherwise, the algorithm sets η˜2←η˜. Using the abovementioned update method instead of formula (32) in algorithm 2, an iterative reinforcement learning algorithm based on the dichotomy of SSP values can be obtained.The Q-value learning curve of the SMDP random shortest path value iterative reinforcement learning algorithm and the random shortest path value iterative reinforcement learning algorithm based on the dichotomy are given, respectively. The simulation experiment results of the algorithm are shown in Figures4 and 5, respectively. The simulation environment of these two algorithms is the same as that of the IVI reinforcement learning algorithm. The learning rate of SSP value iterative reinforcement learning algorithm α, parameters s and o, and the estimated value of the initial optimal average return are the same as the IVI reinforcement learning algorithm, and its search step γ=0.1. The initial estimates η˜1 and η˜2 of the SSP value iterative reinforcement learning algorithm based on the dichotomy are the same as those of the IVI reinforcement learning algorithm based on the dichotomy. The special state in the random shortest path problem is ξ=1.Figure 4
Q-value learning curve of SSP value iterative reinforcement learning algorithm.Figure 5
Q-value learning curve of SSP value iterative reinforcement learning algorithm based on dichotomy.It can be seen from Figures4 and 5 that the SSP value iterative reinforcement learning algorithm and the dichotomy-based SSP value iterative reinforcement learning algorithm both converged within 22,000 iteration steps. Among them, the SSP value iterative reinforcement learning algorithm converges faster than the SSP value iterative reinforcement learning algorithm based on the dichotomy. The SSP value iterative reinforcement learning algorithm has the fastest convergence speed and the best convergence performance. Compared with other semi-Markov reinforcement learning algorithms in this article, the SSP value iterative reinforcement learning algorithm has the highest efficiency in finding the optimal strategy. By comparing Figures 1 and 4, it can be seen that the SSP value iterative reinforcement learning algorithm has improved the convergence performance of the IVI reinforcement learning algorithm. The simulation results in Figures 4 and 5 verify the convergence of the SSP value iterative reinforcement learning algorithm and the SSP value iterative reinforcement learning algorithm based on the dichotomy. Although the effect of using the dichotomy to directly estimate the optimal average return ημ∗ is not as good as the SSP value iterative reinforcement learning algorithm. However, the iterative reinforcement learning algorithm of SSP value based on dichotomy compared with the semi-Markov reinforcement learning algorithm in this paper still has a large degree of improvement in convergence performance.


## 4. Music Education Innovation Model Based on Information Fusion Reinforcement Learning Model

The multimedia system for studying music teaching needs to meet the following basic requirements. First, it needs to build a rich music resource library, which is the core of the multimedia teaching system. Second, for students with a higher start, the system needs to provide high-quality music works for students to appreciate and learn, which will help improve students’ music appreciation level. For students with a low start, the system needs to provide a variety of basic teaching resources to lay a solid foundation for students.There are video resources, document resources, and picture resources in the music multimedia teaching system. The system needs to support online browsing of different types of music resources. In addition, it needs to provide ways to realize the unified browsing of video resources, document resources, and picture resources.In addition, the system needs to support intelligent answering. Traditional teaching systems only support real-time interaction between teachers and students. This interaction mode is limited by time and space, and teachers cannot answer students’ questions if they are not online. In order to solve this defect, the system introduces intelligent Q&A, which can solve most students’ questions and help improve the quality of teaching.Music resources must have a detailed classification system, which is helpful for subsequent location searches and allows students to quickly find music resources of interest.Once the music resources stored in the traditional paper mode are damaged, they cannot be restored. However, the resources stored after informatization must have high reliability and be able to deal with database failures under normal circumstances. The basic requirements can be summarized from the above requirements, such as dynamic management of music resources and music resource classification. At the same time, data backup and data recovery functions need to be introduced to save music resources. The abovementioned analysis is the basis for clarifying the functional requirements of the music multimedia teaching system.The use case diagram of the system management module is shown in Figure6.Figure 6
Use case diagram of system management module.The user can accurately locate the music resources in the system according to the resource keywords to save time. If the user searches for the resources one by one, a lot of time will be wasted. The use case diagram of the music resource management module is shown in Figure7.Figure 7
Use case diagram of music resource management module.Homework management is mainly to examine students’ mastery of each knowledge point. Teachers can post word-based homework online, and students can upload their answers online to the teacher for correction after completion. The use case diagram of the homework management module is shown in Figure8.Figure 8
Use case diagram of homework management module.The purpose of building a music multimedia teaching system in this subject is to realize the sharing of music resources. The physical structure of the system is shown in Figure9.Figure 9
System topology diagram.The music multimedia teaching system is designed and implemented based on the J2EE platform. Developers can assemble commonly used codes in the system to form reusable codes so that they can be reused. The architecture diagram is shown in Figure10.Figure 10
Structure of the system.The music multimedia teaching system is composed of five parts: music resource management, system management, homework management, interactive management, and basic data setting. The functional structure diagram is shown in Figure11.Figure 11
Functional structure diagram.


## 5. Test and Discussion

MATLAB-based expansion test analysis: the specific implementation process of the experiment is to lock the source of the data on the music platform and collect it based on the target of the task.On the basis of the abovementioned analysis, the model proposed in this paper is verified, and the practical effects of the music education innovation system based on the information fusion reinforcement learning model are explored, and the information fusion and teaching effects of the system are, respectively, counted. The method proposed in this paper is compared with the literature in [20]. The results are shown in Tables 1 and 2.Table 1
Evaluation of the effect of music resource information fusion.
NumberThe method of this paperThe method of [20]NumberThe method of this paperThe method of [20]191.4781.911695.7095.27293.6881.131791.5283.11394.4681.631889.0986.11489.5479.001996.9183.27593.8484.772089.6488.15692.4189.932191.2589.53792.1980.552292.3189.91891.9887.332391.2184.25992.7290.382496.3995.971088.0387.572593.6386.971194.8284.142692.4283.821292.9480.342792.2987.451396.4683.992895.4085.421492.6286.472991.2290.571591.1589.303092.9384.80Table 2
Evaluation of the effect of music education innovation.
NumberThe method of this paperThe method of [20]NumberThe method of this paperThe method of [20]177.7482.931671.8682.58289.4992.671779.8080.54377.5082.661891.7278.29485.3789.411985.6290.26572.1184.102087.9288.83680.1386.202176.5384.52773.4485.582281.7288.32882.0786.252385.9488.87985.7679.192490.4584.801072.0680.712576.2784.421188.5691.302687.2189.191290.3485.122782.1586.111379.2392.262881.1488.421472.3385.062971.3780.051585.7085.463071.9679.25From the abovementioned research, it can be seen that the music education innovation system based on the information fusion reinforcement learning model proposed in this paper has a good music resource information fusion effect and teaching effect.


## 6. Conclusion

In modern music education, school education has always appeared as a dominant form and has played an important role. In terms of educational consciousness, music education has always been in a subordinate position, and it has often failed to arouse people’s attention to the role played by music education’s ability to influence. Moreover, music courses have always been marginalized courses or dispensable courses. This is also a kind of partial subject phenomenon caused by the shortcomings of the modern examination system, which has caused no high attention from students to parents to society. In addition, modern music teachers are generally incompletely equipped, which causes modern music education to treat this subject only from the perspective of entertainment. In addition, it has not attracted the attention of the state and society. This paper combines the reinforcement learning model based on information fusion to innovate the way of music education, change the traditional music education model, and improve the actual effect of music education. From the experimental analysis, it can be seen that the music education innovation system based on the information fusion reinforcement learning model proposed in this paper has a good music resource information fusion effect and teaching effect.



---

*Source: 7367418-2022-06-23.xml*