# Multiple Musical Instrument Signal Recognition Based on Convolutional Neural Network

Authors: Lei Lei
Journal: Scientific Programming
 (2022)

---

## Abstract

To improve the accuracy of multi-instrument recognition, based on the basic principles and structure of CNN, a multipitch instrument recognition method based on the convolutional neural network (CNN) is proposed. First of all, the pitch feature detection technology and constantQ transform (CQT) are adopted to extract the signal characteristics of multiple instruments, which are used as the input of the CNN network. Moreover, in order to improve the accuracy of multi-instrument signal recognition, the benchmark recognition model and two-level recognition model are constructed. Finally, the above models are verified by experiments. The results show that the two-level classification model established in this article can accurately identify and classify various musical instruments, and the recognition accuracy is improved most obviously in xylophone. Compared with the benchmark model, the constructed two-level recognition has the highest accuracy and precision, which shows that this model has superior performance and can improve the accuracy of multi-instrument recognition.


---

## Body

## 1. Introduction to Convolutional Neural Network

CNN, a representative algorithm of deep learning, is a kind of feed-forward neural network, which includes convolution calculation and has depth structure [11]. CNN can learn the original data efficiently and quickly, so as to extract the features of the data, which means that it has the ability of representation learning. The specific structure is shown in Figure 1, which is mainly divided into five network layers and belongs to multilayer perceptron (MLP) [12]. The most important steps are convolution calculation and pooling operation.Figure 1
CNN neural network structure.


### 1.1. Convolutional Layer

The convolution formula is as follows:(1)st=xt∗wt=∑τ=−∞τ=+∞xτwt−τ,where st, xt, and wt represent feature mapping, input features, and convolution cores, respectively. If it is two-dimensional matrices, it can be represented as(2)si,j=∑m=0M∑n=0Nwm,nxi+m+wb.In the above formula, the size of convolution kernel isM×N, which is shown in Figure 2 [13–15]. The advantages of convolution operation are mainly reflected in three aspects. Firstly, the realization of parameter sharing helps to reduce the size of the parameter set. Secondly, sparse connection reduces the number of parameters and improves the efficiency, which has certain advantages over full connection. Thirdly, because the same convolution kernel is used, when the value of input eigenmatrix changes, the corresponding result will change at the same position.Figure 2
Example of convolution operation.


### 1.2. Pooling Layer

The pooling layer refers to the output of statistics for a specific region within the input eigenmatrix. Generally, two pooling methods can be adopted, namely, average pooling and maximum pooling. They take the mean and maximum values of the local region as the output, respectively. Except for these two methods, there is a random pooling, which selects neurons with greater probability values.


## 1.1. Convolutional Layer

The convolution formula is as follows:(1)st=xt∗wt=∑τ=−∞τ=+∞xτwt−τ,where st, xt, and wt represent feature mapping, input features, and convolution cores, respectively. If it is two-dimensional matrices, it can be represented as(2)si,j=∑m=0M∑n=0Nwm,nxi+m+wb.In the above formula, the size of convolution kernel isM×N, which is shown in Figure 2 [13–15]. The advantages of convolution operation are mainly reflected in three aspects. Firstly, the realization of parameter sharing helps to reduce the size of the parameter set. Secondly, sparse connection reduces the number of parameters and improves the efficiency, which has certain advantages over full connection. Thirdly, because the same convolution kernel is used, when the value of input eigenmatrix changes, the corresponding result will change at the same position.Figure 2
Example of convolution operation.


## 1.2. Pooling Layer

The pooling layer refers to the output of statistics for a specific region within the input eigenmatrix. Generally, two pooling methods can be adopted, namely, average pooling and maximum pooling. They take the mean and maximum values of the local region as the output, respectively. Except for these two methods, there is a random pooling, which selects neurons with greater probability values.


## 2. The Construction of Multi-Instrument Signal Recognition Model Based on Convolutional Neural Network

### 2.1. Multi-Instrument Signal Feature Extraction

In order to realize the recognition of multi-instrument signals, it is necessary to extract musical instrument signal features first. Conventional instrument signal extraction is usually only for a single instrument, which is relatively simple. It only needs to eliminate the instrument noise and then classify them. But for multi-instrument signals, it not only needs to deal with the noise, but also faces the knowledge of notes of different instruments. In other words, the conventional time-frequency feature extraction, such as MFCC, may not achieve the recognition effect. Therefore, on the basis of signal processing, the essential elements of music, such as pitch, harmony, and other signals, are combined to identify the signals of multiple instruments. The instrument signal characteristics are processed by pitch characteristic detection and constantQ transformation.


#### 2.1.1. Pitch Feature Extraction

The multipitch detection based on a statistical model and spectral decomposition is the main method to extract pitch features. However, considering that an end-to-end neural network may have the problem of overfitting in the display feature extraction, filters are introduced to extract the time-frequency features of musical instrument signals in the primary feature extraction process of convolutional neural network. In other words, the first layer of the convolutional neural network is replaced by the filter, which can greatly reduce the overfitting problem. The specific extraction process is shown in Figure3 [16–18].Figure 3
Pitch feature extraction process.The specific process is as follows:Firstly, the audio frameX is normalized, that is, X⟶X/X2. The audible variables of each frame are standardized. Then, it is divided into Tp segments, and each segment is represented as xt. The number of sampling points is s, which means xt=xt1,xt2,...,xti. To map X, it needs to use the filter banks in the log-frequency domain, including cosine and sine filters, and the total number is np=511. Thus, the logarithmic frequency-time matrix (np×Tp) can be formed, and the log-frequency domain is logfL to logfH. Then, the parameter of sine filter i is shown as follows [19]:(3)wi,sin=sin2πfit1,...,sin2πfits.The parameters of cosine filteri are shown as follows:(4)wi,cos=cos2πfit1,...,cos2πfits.In the above formula,fi=10logfL+ilogfH−logfL/n; according to the normalized amplitude xt=xt1,xt2,...,xti, the position t1,t2,..,ts at each time can be determined.Then, letxt do the inner product calculation with wi,sin and wi,cos. Next, the square and the sum of the two can be calculated. So the output of filter i can be obtained as follows [20–22]:(5)filteri=wi,sinTxt2+wi,cosTxt2.If the number of filters isnp, the corresponding outputs are filter1,...,filternp. Dividing the audio frames into segments, if the number of segments is Tp, the logarithmic frequency-time matrix is np×Tp.Secondly, processing the above matrix, the tensor of logarithmic frequency-time-channel can be obtained after convolution. The first layer of the convolutional network is the matrix obtained in the first step. However, setting up the mapping of the second layer is to do the convolution computation for the logarithmic frequency axis. The step size is set to 3, and the convolution kernel128×1 is selected. The matrix after convolution is mapped to channels, and the tensor of 128×Tp×Ci is obtained, where C1 represents the number of channels.Thirdly, continue the two-dimensional convolution for the tensor obtained in the previous step, so that the logarithmic frequency-channel matrix can be obtained. Mapping to the third layer with the same method, the height of convolution kernel (Tp×C2) is 1; thus, the matrix of 128 × C2 can be obtained, where C2 represents the number of channels.The full connection processing of the matrix obtained in the previous step is performed, and the corresponding pitch recognition vector can be obtained. It is necessary to connect the lines of the matrix in the previous step with the linear classifier. The number of pitch frequency ism1. If the number of valid elements is the same, the vector is 1. If the number is different, the vector is 0.After the frame segmentation is completed, each audio frame is processed based on the above process. The corresponding pitch feature matrix can be obtained. If the pitch frequency in the pitch set isMp, and the number of frame is Np, so the corresponding matrix size is expressed as Mp×Np.


#### 2.1.2. Constant Q Transform

In order to better display the pitch frequency on the spectrum space of DFT or STFT, the constantQ transform (CQT) is adopted in this study to transform the time-frequency of music signal analysis. The specific steps are as follows [23–25]:(1)
Find the spectral kernel matrix corresponding to the octave with the highest frequency.(2)
The corresponding CQT frequency band of the input signalxn is calculated by the DFT transform vector, and the input signal xn is marked as x0n.(3)
Sample the signal.(4)
Calculate the CQT frequency band by the corresponding DFT transform vector of the next octave.(5)
Repeat steps (3) and (4) until the calculation is complete, as shown in Figure4.Figure 4
The method of CQT calculation for each octave.In Figure 4,Gf represents the low-pass filter, and ↓2 represents downsampling with a downsampling factor of 2. Here, the downsampling of xdn is fs/2d (d≥1), and the CQT transform XdCQ of each octave is(6)XdCQ=A∗Xd,whereA∗ represents the conjugate transpose of the complex numerical spectrum kernel matrix, which is usually used to calculate the CQT of octaves.


### 2.2. Construction of Multi-Instrument Signal Recognition Model

#### 2.2.1. Construction of Benchmark Model

First of all, the benchmark model needs to be established, and then the modification and improvement can be achieved on this basis. In this article, combining the convolutional network model designed by liu and Yang, the model can realize automatic music labeling. The data used in this training have a frame-level accuracy label, which is used as a supervisory signal.This model is divided into multiple layers, including the batch standardization layer, convolution layer, pooling layer, etc., and the specific structure is shown in Figure5 [26].Figure 5
Network structure of the benchmark model.Due to the problem of internal covariable offset in the training process of the convolutional network, the batch standardization layer can be used to deal with it. It is necessary to ensure the consistency in the distribution of training and test data, and it is helpful to improve generalization ability. However, when there are many parameters and the number of network layers increases, the data distribution will change after the parameter update. At this time, the difficulty of training will increase. To solve the above problems, the batch standardization can be adopted to adjust the data distribution. It makes the intermediate characteristic data become normal distribution, which is realized by processing the input or output data of the intermediate hidden layer. This is a standardized processing procedures, and its formula is shown as follows [27]:(7)ain=γi×ai−μσ+βi.Here,βi and γi represent translation and zoom factor, respectively; a1 represents initial activation value; μ and σ are as follows, respectively:(8)μ=1m∑k=1mak,k∈S,S=m,σ=1m∑k=1mak−μ2+ε,k∈S,S=m.In the formulas,S represents the neuron set with a size of M, but it has different meanings for different networks. If it is a convolutional network, m actually represents the total number of all activation values formed based on the convolutional kernel channel. For a fully connected network, it represents the number of activation values formed by all instances in a particular batch. ε represents the constants related to training stability.Batch standardization is a key process. The problem of gradient explosion can be solved by adjusting data distribution, because the noise caused by scaling and other operations can help to get the parameters with higher generalization performance. In addition, to improve the efficiency of convergence, a relatively large learning step can be set.Except for the above layers, the function of the convolution layer is to extract the required intermediate features. The maximum pooling layer is an important part, which can realize the function of compression features. And it can reduce the difficulty of calculation. Specifically, the maximum value dividing the pooling area is taken as the output value. ReLu function is adopted in the middle, and the specific form is as follows [28].(9)fx=x,x>0,0,x≤0,where some activation values of output are equal to zero, which makes the network sparse. Compared to the Sigmoid function, a higher convergence speed can be achieved. The output layer is mainly applied, and its form is shown as follows [29]:(10)fx=11+exp−x.Based on the function, the normalization has been achieved. The output value of the instrument recognition model is placed in the range of 0–1, namely, the existence probability of various musical instruments. Then, the binarization method can be adopted to determine instruments’ existing situation. This process depends on the proper threshold. If the threshold is set to 0.5, there is no guarantee for good performance. Therefore, a kind of threshold selection algorithm is designed. It means that the method of maximizing the F1 score of the training set is used to set the threshold value. There are 99 candidates’ threshold values, which are 0.1, 0.2, …, 0.99, respectively.The loss function adopted in the training is binary cross-entropy, and the specific form is shown as follows [30]:(11)l=−∑k=111y^klogyk+1−y^klog1−yk.Here,k represents the specific musical instrument category, and yk and y⌢k represent the identification of each time frame and real label. Considering the imbalance of categories, a certain weight is set for each category, which is expressed as ωk. The specific form is shown as follows [30]:(12)ωk=p¯pk×1−pk1−p¯η.In the formula,η represents the hyperparameter, which is generally valued at 0.3; p¯ represents the mean value of all of pk; and pk represents the proportion occupied by category k. lban is expressed as follows:(13)lban=−∑k=111ωky^klogyk+1−y^klog1−yk.According to the above analysis, the weights need to be set in conjunction with the proportion size occupied by a specific categories of instruments. For example, when the proportion occupied by a specific categories of instruments is low, a higher weight should be set to improve the accuracy of instrument recognition results. So when the frequency of occurrence is not high, the effective recognition even can be ensured. The momentum algorithm is adopted in the calculation, in which weight attenuation factor, learning rate, and batch are2×10−4, 0.01, and 80, respectively.


#### 2.2.2. Multi-Instrument Signal Recognition Based on Two-Level Classification

When multiple instruments are played at the same time, the traditional classification model based on the attention network has a poor recognition effect on harmonic instruments. The main reason is the category imbalance, which means that the difference in the proportion of different categories interferes with the learning of model parameters. Therefore, combined with the basic principle of undersampling or oversampling, the two-level classification model is proposed. This model is mainly divided into the first-level and the second-level convolutional neural network classification models.The first-level classification model takes the constantQ transform matrix as the input feature. Firstly, the instrument families in audio signals are rough classified. The constant Q transform matrix reflects the time-frequency energy distribution of audio signals, and it can be used as an effective feature of rough classification.The second-level classification model is composed of three residual network models with the same architecture. Each residual network model is specially trained to identify various instruments under a certain musical instrument family. There is a special network model for each of the three musical instrument families. The specific process of the two-level classification model is as follows [30, 31].Figure6 shows the network architecture of first-level classification model. From the top to the bottom, there are batch standardization layer, convolution layer, batch standardization layer, convolution layer, convolution layer, batch standardization layer, ReLu layer, maximum pooling layer, convolution layer, and Sigmoid layer.Figure 6
The network structure of the first-level classification model in the two-level classification model.Figure7 shows the residual network model architecture of three same structures in the second-level classification model. From the top to the bottom, there are batch standardization layer, convolution layer, residual block, maximum pooling layer, residual block, convolution layer, maximum pooling layer, batch standardization layer, ReLu layer, convolution layer, and Sigmoid layer.Figure 7
Structure of three residual network models with the same structure in the second-level classification model.The residual block is divided into two parts. The first part includes batch standardization layer, convolution layer, batch standardization layer, ReLu layer, convolution layer, and batch standardization layer. And the second part is a convolution layer. The input of the residual block enters these two parts to obtain the output, and the output of two parts is summed as the final output of the residual block.The residual structure of the residual network can effectively solve the problem of gradient disappearance in deep network. In this article, it is applied to detect and recognize the time-frequency characteristic spectrum, and the intermediate feature extraction of musical instrument recognition is realized successively.The loss function is expressed as(14)ς′ban=−∑k=1Kωky^klogyk+1−y^klog1−yk.In the first-level classification model, theK of equation (14) is valued at 3, representing the three musical instrument family categories of string, wind, and percussion. In the second-level classification model, the K in the string music classification network is valued at 5, representing piano, violin, viola, guitar, and bass. In the wind music classification network, the K is valued at 3, representing saxophone, bassoon, and trumpet. The value of K in the percussion classification network is valued at 3, representing timpani, small drum, and xylophone.


## 2.1. Multi-Instrument Signal Feature Extraction

In order to realize the recognition of multi-instrument signals, it is necessary to extract musical instrument signal features first. Conventional instrument signal extraction is usually only for a single instrument, which is relatively simple. It only needs to eliminate the instrument noise and then classify them. But for multi-instrument signals, it not only needs to deal with the noise, but also faces the knowledge of notes of different instruments. In other words, the conventional time-frequency feature extraction, such as MFCC, may not achieve the recognition effect. Therefore, on the basis of signal processing, the essential elements of music, such as pitch, harmony, and other signals, are combined to identify the signals of multiple instruments. The instrument signal characteristics are processed by pitch characteristic detection and constantQ transformation.


### 2.1.1. Pitch Feature Extraction

The multipitch detection based on a statistical model and spectral decomposition is the main method to extract pitch features. However, considering that an end-to-end neural network may have the problem of overfitting in the display feature extraction, filters are introduced to extract the time-frequency features of musical instrument signals in the primary feature extraction process of convolutional neural network. In other words, the first layer of the convolutional neural network is replaced by the filter, which can greatly reduce the overfitting problem. The specific extraction process is shown in Figure3 [16–18].Figure 3
Pitch feature extraction process.The specific process is as follows:Firstly, the audio frameX is normalized, that is, X⟶X/X2. The audible variables of each frame are standardized. Then, it is divided into Tp segments, and each segment is represented as xt. The number of sampling points is s, which means xt=xt1,xt2,...,xti. To map X, it needs to use the filter banks in the log-frequency domain, including cosine and sine filters, and the total number is np=511. Thus, the logarithmic frequency-time matrix (np×Tp) can be formed, and the log-frequency domain is logfL to logfH. Then, the parameter of sine filter i is shown as follows [19]:(3)wi,sin=sin2πfit1,...,sin2πfits.The parameters of cosine filteri are shown as follows:(4)wi,cos=cos2πfit1,...,cos2πfits.In the above formula,fi=10logfL+ilogfH−logfL/n; according to the normalized amplitude xt=xt1,xt2,...,xti, the position t1,t2,..,ts at each time can be determined.Then, letxt do the inner product calculation with wi,sin and wi,cos. Next, the square and the sum of the two can be calculated. So the output of filter i can be obtained as follows [20–22]:(5)filteri=wi,sinTxt2+wi,cosTxt2.If the number of filters isnp, the corresponding outputs are filter1,...,filternp. Dividing the audio frames into segments, if the number of segments is Tp, the logarithmic frequency-time matrix is np×Tp.Secondly, processing the above matrix, the tensor of logarithmic frequency-time-channel can be obtained after convolution. The first layer of the convolutional network is the matrix obtained in the first step. However, setting up the mapping of the second layer is to do the convolution computation for the logarithmic frequency axis. The step size is set to 3, and the convolution kernel128×1 is selected. The matrix after convolution is mapped to channels, and the tensor of 128×Tp×Ci is obtained, where C1 represents the number of channels.Thirdly, continue the two-dimensional convolution for the tensor obtained in the previous step, so that the logarithmic frequency-channel matrix can be obtained. Mapping to the third layer with the same method, the height of convolution kernel (Tp×C2) is 1; thus, the matrix of 128 × C2 can be obtained, where C2 represents the number of channels.The full connection processing of the matrix obtained in the previous step is performed, and the corresponding pitch recognition vector can be obtained. It is necessary to connect the lines of the matrix in the previous step with the linear classifier. The number of pitch frequency ism1. If the number of valid elements is the same, the vector is 1. If the number is different, the vector is 0.After the frame segmentation is completed, each audio frame is processed based on the above process. The corresponding pitch feature matrix can be obtained. If the pitch frequency in the pitch set isMp, and the number of frame is Np, so the corresponding matrix size is expressed as Mp×Np.


### 2.1.2. Constant Q Transform

In order to better display the pitch frequency on the spectrum space of DFT or STFT, the constantQ transform (CQT) is adopted in this study to transform the time-frequency of music signal analysis. The specific steps are as follows [23–25]:(1)
Find the spectral kernel matrix corresponding to the octave with the highest frequency.(2)
The corresponding CQT frequency band of the input signalxn is calculated by the DFT transform vector, and the input signal xn is marked as x0n.(3)
Sample the signal.(4)
Calculate the CQT frequency band by the corresponding DFT transform vector of the next octave.(5)
Repeat steps (3) and (4) until the calculation is complete, as shown in Figure4.Figure 4
The method of CQT calculation for each octave.In Figure 4,Gf represents the low-pass filter, and ↓2 represents downsampling with a downsampling factor of 2. Here, the downsampling of xdn is fs/2d (d≥1), and the CQT transform XdCQ of each octave is(6)XdCQ=A∗Xd,whereA∗ represents the conjugate transpose of the complex numerical spectrum kernel matrix, which is usually used to calculate the CQT of octaves.


## 2.1.1. Pitch Feature Extraction

The multipitch detection based on a statistical model and spectral decomposition is the main method to extract pitch features. However, considering that an end-to-end neural network may have the problem of overfitting in the display feature extraction, filters are introduced to extract the time-frequency features of musical instrument signals in the primary feature extraction process of convolutional neural network. In other words, the first layer of the convolutional neural network is replaced by the filter, which can greatly reduce the overfitting problem. The specific extraction process is shown in Figure3 [16–18].Figure 3
Pitch feature extraction process.The specific process is as follows:Firstly, the audio frameX is normalized, that is, X⟶X/X2. The audible variables of each frame are standardized. Then, it is divided into Tp segments, and each segment is represented as xt. The number of sampling points is s, which means xt=xt1,xt2,...,xti. To map X, it needs to use the filter banks in the log-frequency domain, including cosine and sine filters, and the total number is np=511. Thus, the logarithmic frequency-time matrix (np×Tp) can be formed, and the log-frequency domain is logfL to logfH. Then, the parameter of sine filter i is shown as follows [19]:(3)wi,sin=sin2πfit1,...,sin2πfits.The parameters of cosine filteri are shown as follows:(4)wi,cos=cos2πfit1,...,cos2πfits.In the above formula,fi=10logfL+ilogfH−logfL/n; according to the normalized amplitude xt=xt1,xt2,...,xti, the position t1,t2,..,ts at each time can be determined.Then, letxt do the inner product calculation with wi,sin and wi,cos. Next, the square and the sum of the two can be calculated. So the output of filter i can be obtained as follows [20–22]:(5)filteri=wi,sinTxt2+wi,cosTxt2.If the number of filters isnp, the corresponding outputs are filter1,...,filternp. Dividing the audio frames into segments, if the number of segments is Tp, the logarithmic frequency-time matrix is np×Tp.Secondly, processing the above matrix, the tensor of logarithmic frequency-time-channel can be obtained after convolution. The first layer of the convolutional network is the matrix obtained in the first step. However, setting up the mapping of the second layer is to do the convolution computation for the logarithmic frequency axis. The step size is set to 3, and the convolution kernel128×1 is selected. The matrix after convolution is mapped to channels, and the tensor of 128×Tp×Ci is obtained, where C1 represents the number of channels.Thirdly, continue the two-dimensional convolution for the tensor obtained in the previous step, so that the logarithmic frequency-channel matrix can be obtained. Mapping to the third layer with the same method, the height of convolution kernel (Tp×C2) is 1; thus, the matrix of 128 × C2 can be obtained, where C2 represents the number of channels.The full connection processing of the matrix obtained in the previous step is performed, and the corresponding pitch recognition vector can be obtained. It is necessary to connect the lines of the matrix in the previous step with the linear classifier. The number of pitch frequency ism1. If the number of valid elements is the same, the vector is 1. If the number is different, the vector is 0.After the frame segmentation is completed, each audio frame is processed based on the above process. The corresponding pitch feature matrix can be obtained. If the pitch frequency in the pitch set isMp, and the number of frame is Np, so the corresponding matrix size is expressed as Mp×Np.


## 2.1.2. Constant Q Transform

In order to better display the pitch frequency on the spectrum space of DFT or STFT, the constantQ transform (CQT) is adopted in this study to transform the time-frequency of music signal analysis. The specific steps are as follows [23–25]:(1)
Find the spectral kernel matrix corresponding to the octave with the highest frequency.(2)
The corresponding CQT frequency band of the input signalxn is calculated by the DFT transform vector, and the input signal xn is marked as x0n.(3)
Sample the signal.(4)
Calculate the CQT frequency band by the corresponding DFT transform vector of the next octave.(5)
Repeat steps (3) and (4) until the calculation is complete, as shown in Figure4.Figure 4
The method of CQT calculation for each octave.In Figure 4,Gf represents the low-pass filter, and ↓2 represents downsampling with a downsampling factor of 2. Here, the downsampling of xdn is fs/2d (d≥1), and the CQT transform XdCQ of each octave is(6)XdCQ=A∗Xd,whereA∗ represents the conjugate transpose of the complex numerical spectrum kernel matrix, which is usually used to calculate the CQT of octaves.


## 2.2. Construction of Multi-Instrument Signal Recognition Model

### 2.2.1. Construction of Benchmark Model

First of all, the benchmark model needs to be established, and then the modification and improvement can be achieved on this basis. In this article, combining the convolutional network model designed by liu and Yang, the model can realize automatic music labeling. The data used in this training have a frame-level accuracy label, which is used as a supervisory signal.This model is divided into multiple layers, including the batch standardization layer, convolution layer, pooling layer, etc., and the specific structure is shown in Figure5 [26].Figure 5
Network structure of the benchmark model.Due to the problem of internal covariable offset in the training process of the convolutional network, the batch standardization layer can be used to deal with it. It is necessary to ensure the consistency in the distribution of training and test data, and it is helpful to improve generalization ability. However, when there are many parameters and the number of network layers increases, the data distribution will change after the parameter update. At this time, the difficulty of training will increase. To solve the above problems, the batch standardization can be adopted to adjust the data distribution. It makes the intermediate characteristic data become normal distribution, which is realized by processing the input or output data of the intermediate hidden layer. This is a standardized processing procedures, and its formula is shown as follows [27]:(7)ain=γi×ai−μσ+βi.Here,βi and γi represent translation and zoom factor, respectively; a1 represents initial activation value; μ and σ are as follows, respectively:(8)μ=1m∑k=1mak,k∈S,S=m,σ=1m∑k=1mak−μ2+ε,k∈S,S=m.In the formulas,S represents the neuron set with a size of M, but it has different meanings for different networks. If it is a convolutional network, m actually represents the total number of all activation values formed based on the convolutional kernel channel. For a fully connected network, it represents the number of activation values formed by all instances in a particular batch. ε represents the constants related to training stability.Batch standardization is a key process. The problem of gradient explosion can be solved by adjusting data distribution, because the noise caused by scaling and other operations can help to get the parameters with higher generalization performance. In addition, to improve the efficiency of convergence, a relatively large learning step can be set.Except for the above layers, the function of the convolution layer is to extract the required intermediate features. The maximum pooling layer is an important part, which can realize the function of compression features. And it can reduce the difficulty of calculation. Specifically, the maximum value dividing the pooling area is taken as the output value. ReLu function is adopted in the middle, and the specific form is as follows [28].(9)fx=x,x>0,0,x≤0,where some activation values of output are equal to zero, which makes the network sparse. Compared to the Sigmoid function, a higher convergence speed can be achieved. The output layer is mainly applied, and its form is shown as follows [29]:(10)fx=11+exp−x.Based on the function, the normalization has been achieved. The output value of the instrument recognition model is placed in the range of 0–1, namely, the existence probability of various musical instruments. Then, the binarization method can be adopted to determine instruments’ existing situation. This process depends on the proper threshold. If the threshold is set to 0.5, there is no guarantee for good performance. Therefore, a kind of threshold selection algorithm is designed. It means that the method of maximizing the F1 score of the training set is used to set the threshold value. There are 99 candidates’ threshold values, which are 0.1, 0.2, …, 0.99, respectively.The loss function adopted in the training is binary cross-entropy, and the specific form is shown as follows [30]:(11)l=−∑k=111y^klogyk+1−y^klog1−yk.Here,k represents the specific musical instrument category, and yk and y⌢k represent the identification of each time frame and real label. Considering the imbalance of categories, a certain weight is set for each category, which is expressed as ωk. The specific form is shown as follows [30]:(12)ωk=p¯pk×1−pk1−p¯η.In the formula,η represents the hyperparameter, which is generally valued at 0.3; p¯ represents the mean value of all of pk; and pk represents the proportion occupied by category k. lban is expressed as follows:(13)lban=−∑k=111ωky^klogyk+1−y^klog1−yk.According to the above analysis, the weights need to be set in conjunction with the proportion size occupied by a specific categories of instruments. For example, when the proportion occupied by a specific categories of instruments is low, a higher weight should be set to improve the accuracy of instrument recognition results. So when the frequency of occurrence is not high, the effective recognition even can be ensured. The momentum algorithm is adopted in the calculation, in which weight attenuation factor, learning rate, and batch are2×10−4, 0.01, and 80, respectively.


### 2.2.2. Multi-Instrument Signal Recognition Based on Two-Level Classification

When multiple instruments are played at the same time, the traditional classification model based on the attention network has a poor recognition effect on harmonic instruments. The main reason is the category imbalance, which means that the difference in the proportion of different categories interferes with the learning of model parameters. Therefore, combined with the basic principle of undersampling or oversampling, the two-level classification model is proposed. This model is mainly divided into the first-level and the second-level convolutional neural network classification models.The first-level classification model takes the constantQ transform matrix as the input feature. Firstly, the instrument families in audio signals are rough classified. The constant Q transform matrix reflects the time-frequency energy distribution of audio signals, and it can be used as an effective feature of rough classification.The second-level classification model is composed of three residual network models with the same architecture. Each residual network model is specially trained to identify various instruments under a certain musical instrument family. There is a special network model for each of the three musical instrument families. The specific process of the two-level classification model is as follows [30, 31].Figure6 shows the network architecture of first-level classification model. From the top to the bottom, there are batch standardization layer, convolution layer, batch standardization layer, convolution layer, convolution layer, batch standardization layer, ReLu layer, maximum pooling layer, convolution layer, and Sigmoid layer.Figure 6
The network structure of the first-level classification model in the two-level classification model.Figure7 shows the residual network model architecture of three same structures in the second-level classification model. From the top to the bottom, there are batch standardization layer, convolution layer, residual block, maximum pooling layer, residual block, convolution layer, maximum pooling layer, batch standardization layer, ReLu layer, convolution layer, and Sigmoid layer.Figure 7
Structure of three residual network models with the same structure in the second-level classification model.The residual block is divided into two parts. The first part includes batch standardization layer, convolution layer, batch standardization layer, ReLu layer, convolution layer, and batch standardization layer. And the second part is a convolution layer. The input of the residual block enters these two parts to obtain the output, and the output of two parts is summed as the final output of the residual block.The residual structure of the residual network can effectively solve the problem of gradient disappearance in deep network. In this article, it is applied to detect and recognize the time-frequency characteristic spectrum, and the intermediate feature extraction of musical instrument recognition is realized successively.The loss function is expressed as(14)ς′ban=−∑k=1Kωky^klogyk+1−y^klog1−yk.In the first-level classification model, theK of equation (14) is valued at 3, representing the three musical instrument family categories of string, wind, and percussion. In the second-level classification model, the K in the string music classification network is valued at 5, representing piano, violin, viola, guitar, and bass. In the wind music classification network, the K is valued at 3, representing saxophone, bassoon, and trumpet. The value of K in the percussion classification network is valued at 3, representing timpani, small drum, and xylophone.


## 2.2.1. Construction of Benchmark Model

First of all, the benchmark model needs to be established, and then the modification and improvement can be achieved on this basis. In this article, combining the convolutional network model designed by liu and Yang, the model can realize automatic music labeling. The data used in this training have a frame-level accuracy label, which is used as a supervisory signal.This model is divided into multiple layers, including the batch standardization layer, convolution layer, pooling layer, etc., and the specific structure is shown in Figure5 [26].Figure 5
Network structure of the benchmark model.Due to the problem of internal covariable offset in the training process of the convolutional network, the batch standardization layer can be used to deal with it. It is necessary to ensure the consistency in the distribution of training and test data, and it is helpful to improve generalization ability. However, when there are many parameters and the number of network layers increases, the data distribution will change after the parameter update. At this time, the difficulty of training will increase. To solve the above problems, the batch standardization can be adopted to adjust the data distribution. It makes the intermediate characteristic data become normal distribution, which is realized by processing the input or output data of the intermediate hidden layer. This is a standardized processing procedures, and its formula is shown as follows [27]:(7)ain=γi×ai−μσ+βi.Here,βi and γi represent translation and zoom factor, respectively; a1 represents initial activation value; μ and σ are as follows, respectively:(8)μ=1m∑k=1mak,k∈S,S=m,σ=1m∑k=1mak−μ2+ε,k∈S,S=m.In the formulas,S represents the neuron set with a size of M, but it has different meanings for different networks. If it is a convolutional network, m actually represents the total number of all activation values formed based on the convolutional kernel channel. For a fully connected network, it represents the number of activation values formed by all instances in a particular batch. ε represents the constants related to training stability.Batch standardization is a key process. The problem of gradient explosion can be solved by adjusting data distribution, because the noise caused by scaling and other operations can help to get the parameters with higher generalization performance. In addition, to improve the efficiency of convergence, a relatively large learning step can be set.Except for the above layers, the function of the convolution layer is to extract the required intermediate features. The maximum pooling layer is an important part, which can realize the function of compression features. And it can reduce the difficulty of calculation. Specifically, the maximum value dividing the pooling area is taken as the output value. ReLu function is adopted in the middle, and the specific form is as follows [28].(9)fx=x,x>0,0,x≤0,where some activation values of output are equal to zero, which makes the network sparse. Compared to the Sigmoid function, a higher convergence speed can be achieved. The output layer is mainly applied, and its form is shown as follows [29]:(10)fx=11+exp−x.Based on the function, the normalization has been achieved. The output value of the instrument recognition model is placed in the range of 0–1, namely, the existence probability of various musical instruments. Then, the binarization method can be adopted to determine instruments’ existing situation. This process depends on the proper threshold. If the threshold is set to 0.5, there is no guarantee for good performance. Therefore, a kind of threshold selection algorithm is designed. It means that the method of maximizing the F1 score of the training set is used to set the threshold value. There are 99 candidates’ threshold values, which are 0.1, 0.2, …, 0.99, respectively.The loss function adopted in the training is binary cross-entropy, and the specific form is shown as follows [30]:(11)l=−∑k=111y^klogyk+1−y^klog1−yk.Here,k represents the specific musical instrument category, and yk and y⌢k represent the identification of each time frame and real label. Considering the imbalance of categories, a certain weight is set for each category, which is expressed as ωk. The specific form is shown as follows [30]:(12)ωk=p¯pk×1−pk1−p¯η.In the formula,η represents the hyperparameter, which is generally valued at 0.3; p¯ represents the mean value of all of pk; and pk represents the proportion occupied by category k. lban is expressed as follows:(13)lban=−∑k=111ωky^klogyk+1−y^klog1−yk.According to the above analysis, the weights need to be set in conjunction with the proportion size occupied by a specific categories of instruments. For example, when the proportion occupied by a specific categories of instruments is low, a higher weight should be set to improve the accuracy of instrument recognition results. So when the frequency of occurrence is not high, the effective recognition even can be ensured. The momentum algorithm is adopted in the calculation, in which weight attenuation factor, learning rate, and batch are2×10−4, 0.01, and 80, respectively.


## 2.2.2. Multi-Instrument Signal Recognition Based on Two-Level Classification

When multiple instruments are played at the same time, the traditional classification model based on the attention network has a poor recognition effect on harmonic instruments. The main reason is the category imbalance, which means that the difference in the proportion of different categories interferes with the learning of model parameters. Therefore, combined with the basic principle of undersampling or oversampling, the two-level classification model is proposed. This model is mainly divided into the first-level and the second-level convolutional neural network classification models.The first-level classification model takes the constantQ transform matrix as the input feature. Firstly, the instrument families in audio signals are rough classified. The constant Q transform matrix reflects the time-frequency energy distribution of audio signals, and it can be used as an effective feature of rough classification.The second-level classification model is composed of three residual network models with the same architecture. Each residual network model is specially trained to identify various instruments under a certain musical instrument family. There is a special network model for each of the three musical instrument families. The specific process of the two-level classification model is as follows [30, 31].Figure6 shows the network architecture of first-level classification model. From the top to the bottom, there are batch standardization layer, convolution layer, batch standardization layer, convolution layer, convolution layer, batch standardization layer, ReLu layer, maximum pooling layer, convolution layer, and Sigmoid layer.Figure 6
The network structure of the first-level classification model in the two-level classification model.Figure7 shows the residual network model architecture of three same structures in the second-level classification model. From the top to the bottom, there are batch standardization layer, convolution layer, residual block, maximum pooling layer, residual block, convolution layer, maximum pooling layer, batch standardization layer, ReLu layer, convolution layer, and Sigmoid layer.Figure 7
Structure of three residual network models with the same structure in the second-level classification model.The residual block is divided into two parts. The first part includes batch standardization layer, convolution layer, batch standardization layer, ReLu layer, convolution layer, and batch standardization layer. And the second part is a convolution layer. The input of the residual block enters these two parts to obtain the output, and the output of two parts is summed as the final output of the residual block.The residual structure of the residual network can effectively solve the problem of gradient disappearance in deep network. In this article, it is applied to detect and recognize the time-frequency characteristic spectrum, and the intermediate feature extraction of musical instrument recognition is realized successively.The loss function is expressed as(14)ς′ban=−∑k=1Kωky^klogyk+1−y^klog1−yk.In the first-level classification model, theK of equation (14) is valued at 3, representing the three musical instrument family categories of string, wind, and percussion. In the second-level classification model, the K in the string music classification network is valued at 5, representing piano, violin, viola, guitar, and bass. In the wind music classification network, the K is valued at 3, representing saxophone, bassoon, and trumpet. The value of K in the percussion classification network is valued at 3, representing timpani, small drum, and xylophone.


## 3. Simulation Verification

### 3.1. Experimental Environment

In order to achieve better experimental results, Intel i7-7800X CPU is selected as the hardware system in this experiment. The main frequency is 3.5 GHz, and the farce frequency is 4.0 GHz. It uses 6 cores and 12 threads. The memory is 16 GB, and the graphics card is an NVIDIA GTX 2080 dual-channel GPU.Software system: Ubuntu 16.04, 64 bit operating system, Anaconda3-4.4.0, deep learning framework PyTorch0.4.1, and acceleration module CUDA 10.0.


### 3.2. Dataset Sources

At present, the commonly used dataset includes Bach10 dataset, MedleyDB dataset, and MIXING SECRETS dataset. Among them, Bach10 dataset includes ten large choral works by J.S.Bach, each of which contains four monophonic parts. The audio recordings of each monophonic part are performed by violin, clarinet, saxophone, and bassoon; the MedleyDB dataset consists of 122 songs, in which 108 songs are vocal and instrumental melodies; the MIXING SECRETS dataset contains 258 multitrack audio songs, and there are a variety of music genres involved. However, the scale of the above three public datasets is still not large, and there is no note label in the MIXING SECRETS dataset, only instrument label. Moreover, there are 14 songs in the Medley DB dataset and no note label. In order to solve the above problems, the label annotation information of the MIDI score in an open-source music platform is aligned to the original audio by means of self-built dataset, and then, it is manually calibrated by people with professional music background. Finally, there are 307 useable extended datasets obtained, including various musical instruments and music types, and every frame has the annotation label.


### 3.3. Processing of Pitch Feature Matrix

Musical instruments of the same family have certain similarities in pitch range. In order to better identify, the energy ratio of harmonics needs to be considered. Therefore, based on the extracted pitch feature matrix, lines 7–94 of the extracted matrix are expressed asY1. According to available information, the matrix is sparse. Then, the fundamental frequency position value within Y1 is moved to 12 grids 212/12=2 distance from it. Thus, the matrix Y2 is formed. Using the same way to move to 19 grid 219/12=3, 24 grid 224/12=4, 28 grid 228/12=5, and 31 grid 231/12=6 in turn, the matrix is respectively represented as Y3, Y4, Y5, and Y6.The basic form of harmonic sequence matrix is as follows:(15)Sn=Y1+Y2+…+Yn.The matrix actually represents a sequence combination of fundamental frequencies and corresponding harmonics. Therefore, when determining the meaning ofY6, S1–S6 matrices can be obtained, which are the input feature of the CNN.


### 3.4. Experimental Results and Analysis

#### 3.4.1. Multi-Instrument Recognition Results under Benchmark Model

There are ten kinds of musical instruments, which are divided into three categories: percussion instruments, string instruments, and wind instruments. The percussion instruments are xylophone, timpani, trumpet, and the string instruments are guitar, piano, bass, viola, and violin. In addition, the wind instruments are bassoon and saxophone. In the experiment, an appropriate experimental environment should be configured first, which is basically consistent with the previous pitch feature extraction experiment. The dataset is divided into two parts, namely, training set and test set. The ratio of the two parts is 9 : 1. Moreover, the possibility of the inexistence and unlabeled instruments in the training set should be considered.The extracted constantQ transform matrix XCQ88×165 is processed, which is spliced with S1–S6. The corresponding input matrices can be obtained, which can be represented as I1–I6, namely, the harmonic mapping matrix. Then, they are input into the model, and the corresponding class-time series matrix can be obtained. The feature changes in the benchmark model are shown in Table 1.Table 1
Change process of the feature size in the benchmark model.
Input sizeOperationOutput size176 × 165 × 12 × 1 Convolution kernel, 352 channels352 × 164 × 1352 × 164 × 13 × 1 Maximum pooling352 × 54 × 1352 × 54 × 13 × 1 Convolution kernel, 704 channels704 × 52 × 1704 × 52 × 13 × 1 Channels704 × 17 × 1704 × 17 × 12 × 1 Channels, 704 channels704 × 8 × 1704 × 8 × 11 × 1 Channels, 11 channels11 × 8The experimental results are obtained according to Table2. The instrument-type recognition results are evaluated through F1. In the total number of instrument recognition, there are three cases of unrecognized, misrecognized, and correct recognition. It can be seen that the overall accuracy can identify the proportion of accuracy times.Table 2
F1 and overall accuracy of ten instruments under the benchmark model (using estimated pitch).
Harmonic mapping matrix orderPianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyI10.880.880.840.880.820.820.760.730.770.800.77I20.890.890.840.890.840.830.780.750.730.810.77I30.900.90838.000 8920.830 8360.780.760.800.740.77I40.900.40832.000.890.840.840.790.750.750.790.77I50.890.890.840.890.850.850.800.770.800.760.77I60.890.890.830.890.850.850.040.770.770.790.78The real pitch label matrix is adopted to design the harmonic mapping matrixIng (n is 1–6). At this time, the values in Y1 are all accurate. According to the information in Tables 2 and 3, it can be clearly seen that compared with the estimated pitch labels, the harmonic mapping matrix obtained by using real labels can achieve higher overall accuracy and F1 value. In addition, compared with xylophone and timpani, the recognition results of estimated and real pitch of a small drum are basically the same, which is mainly related to the unfixed pitch. In this study, the pitch features are mainly used to recognize the musical instruments. So the recognition scores of different types of musical instruments are different. Compared with percussion instruments, the recognition scores of orchestral instruments are higher, which verifies the effectiveness of pitch feature extraction.Table 3
F1 scores and overall accuracy of ten instruments under the benchmark model (using real pitch).
Harmonic mapping matrix orderPianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyI1g0.900.900.850.890.840.830.770.740.770.830.80I2g0.910.910.860.890.850.830.800.760.750.850.80I3g0.910.910.850.910.850.840.790.770.760.840.80I4g0.910.910.850.900.850.860.810.780.770.850.81I5g0.910.910.840.900.860.870.820.790.800.810.80I6g0.910.910.850.900.860.870.770.780.790.820.80


#### 3.4.2. Multi-Instrument Classification Results under Two-Level Recognition

Firstly, the configuration of the experimental environment is consistent with the previous section. The momentum algorithm (0.9) is adopted, where the weight attenuation factor, learning rate, and batch are 2 ∗ 10−4, 0.05, and 60, respectively. In this experiment, a tensor is input, including I1–I6. Furthermore, the output result is class-time series matrix. The specific characteristic changes are shown in Table 4.Table 4
Changes of feature size in the classification model based on the attention network.
Input sizeOperationOutput size176 × 165 × 62 × 1 Convolution kernel, 352 channels352 × 164 × 6352 × 164 × 63 × 1 Maximum pooling352 × 54 × 6352 × 52 × 63 × 1 Convolution kernel, 704 channels704 × 52 × 6704 × 52 × 63 × 1 Channels704 × 17 × 6704 × 17 × 62 × 1 Channels, 11 channels11 × 8 × 6704 × 17 × 6Attention subnetSix attention weights11 × 8 × 6The sum using the weighting of attention weight11 × 8I3 and I5 as the benchmark model of input are represented as BI3 and BI5, respectively. The classification model is represented as MA. According to the information in Table 5, it can be seen that compared with the previously adopted benchmark model, a higher overall accuracy is achieved by adopting the attention network model, and the recognition scores of all instruments except xylophone are improved.Table 5
F1 and overall accuracy and comparison of ten musical instruments based on the attention network classification model.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyBI30.900.900.840.890.830.840.780.760.810.730.77BI50.890.890.840.890.850.850.800.770.760.880.77MA0.910.900.850.910.860.860.810.740.800.860.83Based on the analysis of the above phenomenon, it is found that the attention network actually is to set the appropriate weights for the intermediate feature graph. Compared with the other types of musical instruments, the melodic instrument features are more conducive to recognition, which means the weights are higher. The common melodic instruments are piano and guitar, and the xylophone is rarely used as a melodic instrument. Therefore, if it exists at the same time with other instruments, the characteristics that need to be recognized can be easily masked. In this perspective, after adding the attention network, it is beneficial for melody instrument recognition, which is helpful to improve the overall accuracy. However, it is not possible to improve all instrument recognition scores, which needs to be further studied.In this article, the constantQ transform matrix of the first-level classification model is used as input and output instrument family-time series matrix. In the second-level classification model, the third-order harmonic mapping matrix I3, the fifth-order harmonic mapping matrix I5, and the sixth-order harmonic mapping matrix I6 are used as the input features of the string music classification network, wind music classification network, and percussion music classification network, respectively. Then, the output of the three networks is summarized to obtain the final instrument class-time series matrix. The recognition scores and overall accuracy of various musical instruments in the two-level classification model (MT) are obtained, and the comparison is shown in Table 6.Table 6
Comparison of F1 and overall accuracy of ten musical instruments in the two-level classification model.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyBI30.900.900.840.890.830.840.780.760.790.750.77BI50.890.890.840.890.850.850.800.770.710.830.77MA0.910.900.850.910.860.860.810.740.750.910.83MT0.910.900.850.910.860.870.820.810.830.890.86After comprehensively analyzing the charts, what can be found is that the recognition scores of most musical instruments are improved, especially xylophone. And it can be seen that the two-level classification model proposed in this article can balance the classification of musical instruments well, and the overall accuracy is further improved.Figure8 is the recognition effect diagram of the string music classification network. The upper part (a) represents the real label, the lower part (b) represents the recognition result, and the black part represents the existence of musical instruments.Figure 8
Recognition effect of string music classification network in the second-level classification model.
(a)(b)As can be seen intuitively from the figure above, piano and violin can be accurately identified, while there is confusion in the viola, and the recognition accuracy needs to be improved.Figure9 is the identification effect diagram of the wind music classification network. As can be seen from the picture, the trumpet can be identified accurately, while there are confusions in the the other two instruments, and the recognition accuracy needs to be improved.Figure 9
Identification effect of the wind music classification network in the second-level classification model.
(a)(b)Figure10 is the identification effect diagram of the percussion music classification network. It can be clearly seen from the figure that all three musical instruments have been accurately identified, which means that there are obvious differences between these three musical instruments, so that they can be well identified and classified.Figure 10
Recognition effect of the percussion classification network in the second-level classification model.
(a)(b)The comprehensive analysis shows that the two-level classification model constructed in this article has the best comprehensive performance, and it has the more accurate recognition effect.To further verify the effectiveness of the proposed method, the experiment compares the accuracy of the proposed model with that of the existing duets, trios, and quartet, and the obtained comparison results are shown in Table7.Table 7
Comparison between the proposed model and existing methods.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyMA0.9120.9030.8490.9080.8550.8590.8090.7430.7620.8890.825Bipolar classification model0.9130.9010.8530.9070.8630.8660.8220.8110.8340.8800.857Duet0.8270.881—0.829——————0.841Trio0.6830.825—0.828——————0.778Quartet0.5730.799—0.791——————0.731As can be seen from the above table, compared with the other three methods, the recognition accuracy of the two-level classification model proposed in this article is as high as 85.7%. The recognition accuracy of duet is 84.1%. There are 77.8% for trio, and there are 73.1% for quartet. The method proposed in this article is much higher than the other three methods, which shows that the method proposed in this article has a higher recognition accuracy and better performance.


## 3.1. Experimental Environment

In order to achieve better experimental results, Intel i7-7800X CPU is selected as the hardware system in this experiment. The main frequency is 3.5 GHz, and the farce frequency is 4.0 GHz. It uses 6 cores and 12 threads. The memory is 16 GB, and the graphics card is an NVIDIA GTX 2080 dual-channel GPU.Software system: Ubuntu 16.04, 64 bit operating system, Anaconda3-4.4.0, deep learning framework PyTorch0.4.1, and acceleration module CUDA 10.0.


## 3.2. Dataset Sources

At present, the commonly used dataset includes Bach10 dataset, MedleyDB dataset, and MIXING SECRETS dataset. Among them, Bach10 dataset includes ten large choral works by J.S.Bach, each of which contains four monophonic parts. The audio recordings of each monophonic part are performed by violin, clarinet, saxophone, and bassoon; the MedleyDB dataset consists of 122 songs, in which 108 songs are vocal and instrumental melodies; the MIXING SECRETS dataset contains 258 multitrack audio songs, and there are a variety of music genres involved. However, the scale of the above three public datasets is still not large, and there is no note label in the MIXING SECRETS dataset, only instrument label. Moreover, there are 14 songs in the Medley DB dataset and no note label. In order to solve the above problems, the label annotation information of the MIDI score in an open-source music platform is aligned to the original audio by means of self-built dataset, and then, it is manually calibrated by people with professional music background. Finally, there are 307 useable extended datasets obtained, including various musical instruments and music types, and every frame has the annotation label.


## 3.3. Processing of Pitch Feature Matrix

Musical instruments of the same family have certain similarities in pitch range. In order to better identify, the energy ratio of harmonics needs to be considered. Therefore, based on the extracted pitch feature matrix, lines 7–94 of the extracted matrix are expressed asY1. According to available information, the matrix is sparse. Then, the fundamental frequency position value within Y1 is moved to 12 grids 212/12=2 distance from it. Thus, the matrix Y2 is formed. Using the same way to move to 19 grid 219/12=3, 24 grid 224/12=4, 28 grid 228/12=5, and 31 grid 231/12=6 in turn, the matrix is respectively represented as Y3, Y4, Y5, and Y6.The basic form of harmonic sequence matrix is as follows:(15)Sn=Y1+Y2+…+Yn.The matrix actually represents a sequence combination of fundamental frequencies and corresponding harmonics. Therefore, when determining the meaning ofY6, S1–S6 matrices can be obtained, which are the input feature of the CNN.


## 3.4. Experimental Results and Analysis

### 3.4.1. Multi-Instrument Recognition Results under Benchmark Model

There are ten kinds of musical instruments, which are divided into three categories: percussion instruments, string instruments, and wind instruments. The percussion instruments are xylophone, timpani, trumpet, and the string instruments are guitar, piano, bass, viola, and violin. In addition, the wind instruments are bassoon and saxophone. In the experiment, an appropriate experimental environment should be configured first, which is basically consistent with the previous pitch feature extraction experiment. The dataset is divided into two parts, namely, training set and test set. The ratio of the two parts is 9 : 1. Moreover, the possibility of the inexistence and unlabeled instruments in the training set should be considered.The extracted constantQ transform matrix XCQ88×165 is processed, which is spliced with S1–S6. The corresponding input matrices can be obtained, which can be represented as I1–I6, namely, the harmonic mapping matrix. Then, they are input into the model, and the corresponding class-time series matrix can be obtained. The feature changes in the benchmark model are shown in Table 1.Table 1
Change process of the feature size in the benchmark model.
Input sizeOperationOutput size176 × 165 × 12 × 1 Convolution kernel, 352 channels352 × 164 × 1352 × 164 × 13 × 1 Maximum pooling352 × 54 × 1352 × 54 × 13 × 1 Convolution kernel, 704 channels704 × 52 × 1704 × 52 × 13 × 1 Channels704 × 17 × 1704 × 17 × 12 × 1 Channels, 704 channels704 × 8 × 1704 × 8 × 11 × 1 Channels, 11 channels11 × 8The experimental results are obtained according to Table2. The instrument-type recognition results are evaluated through F1. In the total number of instrument recognition, there are three cases of unrecognized, misrecognized, and correct recognition. It can be seen that the overall accuracy can identify the proportion of accuracy times.Table 2
F1 and overall accuracy of ten instruments under the benchmark model (using estimated pitch).
Harmonic mapping matrix orderPianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyI10.880.880.840.880.820.820.760.730.770.800.77I20.890.890.840.890.840.830.780.750.730.810.77I30.900.90838.000 8920.830 8360.780.760.800.740.77I40.900.40832.000.890.840.840.790.750.750.790.77I50.890.890.840.890.850.850.800.770.800.760.77I60.890.890.830.890.850.850.040.770.770.790.78The real pitch label matrix is adopted to design the harmonic mapping matrixIng (n is 1–6). At this time, the values in Y1 are all accurate. According to the information in Tables 2 and 3, it can be clearly seen that compared with the estimated pitch labels, the harmonic mapping matrix obtained by using real labels can achieve higher overall accuracy and F1 value. In addition, compared with xylophone and timpani, the recognition results of estimated and real pitch of a small drum are basically the same, which is mainly related to the unfixed pitch. In this study, the pitch features are mainly used to recognize the musical instruments. So the recognition scores of different types of musical instruments are different. Compared with percussion instruments, the recognition scores of orchestral instruments are higher, which verifies the effectiveness of pitch feature extraction.Table 3
F1 scores and overall accuracy of ten instruments under the benchmark model (using real pitch).
Harmonic mapping matrix orderPianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyI1g0.900.900.850.890.840.830.770.740.770.830.80I2g0.910.910.860.890.850.830.800.760.750.850.80I3g0.910.910.850.910.850.840.790.770.760.840.80I4g0.910.910.850.900.850.860.810.780.770.850.81I5g0.910.910.840.900.860.870.820.790.800.810.80I6g0.910.910.850.900.860.870.770.780.790.820.80


### 3.4.2. Multi-Instrument Classification Results under Two-Level Recognition

Firstly, the configuration of the experimental environment is consistent with the previous section. The momentum algorithm (0.9) is adopted, where the weight attenuation factor, learning rate, and batch are 2 ∗ 10−4, 0.05, and 60, respectively. In this experiment, a tensor is input, including I1–I6. Furthermore, the output result is class-time series matrix. The specific characteristic changes are shown in Table 4.Table 4
Changes of feature size in the classification model based on the attention network.
Input sizeOperationOutput size176 × 165 × 62 × 1 Convolution kernel, 352 channels352 × 164 × 6352 × 164 × 63 × 1 Maximum pooling352 × 54 × 6352 × 52 × 63 × 1 Convolution kernel, 704 channels704 × 52 × 6704 × 52 × 63 × 1 Channels704 × 17 × 6704 × 17 × 62 × 1 Channels, 11 channels11 × 8 × 6704 × 17 × 6Attention subnetSix attention weights11 × 8 × 6The sum using the weighting of attention weight11 × 8I3 and I5 as the benchmark model of input are represented as BI3 and BI5, respectively. The classification model is represented as MA. According to the information in Table 5, it can be seen that compared with the previously adopted benchmark model, a higher overall accuracy is achieved by adopting the attention network model, and the recognition scores of all instruments except xylophone are improved.Table 5
F1 and overall accuracy and comparison of ten musical instruments based on the attention network classification model.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyBI30.900.900.840.890.830.840.780.760.810.730.77BI50.890.890.840.890.850.850.800.770.760.880.77MA0.910.900.850.910.860.860.810.740.800.860.83Based on the analysis of the above phenomenon, it is found that the attention network actually is to set the appropriate weights for the intermediate feature graph. Compared with the other types of musical instruments, the melodic instrument features are more conducive to recognition, which means the weights are higher. The common melodic instruments are piano and guitar, and the xylophone is rarely used as a melodic instrument. Therefore, if it exists at the same time with other instruments, the characteristics that need to be recognized can be easily masked. In this perspective, after adding the attention network, it is beneficial for melody instrument recognition, which is helpful to improve the overall accuracy. However, it is not possible to improve all instrument recognition scores, which needs to be further studied.In this article, the constantQ transform matrix of the first-level classification model is used as input and output instrument family-time series matrix. In the second-level classification model, the third-order harmonic mapping matrix I3, the fifth-order harmonic mapping matrix I5, and the sixth-order harmonic mapping matrix I6 are used as the input features of the string music classification network, wind music classification network, and percussion music classification network, respectively. Then, the output of the three networks is summarized to obtain the final instrument class-time series matrix. The recognition scores and overall accuracy of various musical instruments in the two-level classification model (MT) are obtained, and the comparison is shown in Table 6.Table 6
Comparison of F1 and overall accuracy of ten musical instruments in the two-level classification model.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyBI30.900.900.840.890.830.840.780.760.790.750.77BI50.890.890.840.890.850.850.800.770.710.830.77MA0.910.900.850.910.860.860.810.740.750.910.83MT0.910.900.850.910.860.870.820.810.830.890.86After comprehensively analyzing the charts, what can be found is that the recognition scores of most musical instruments are improved, especially xylophone. And it can be seen that the two-level classification model proposed in this article can balance the classification of musical instruments well, and the overall accuracy is further improved.Figure8 is the recognition effect diagram of the string music classification network. The upper part (a) represents the real label, the lower part (b) represents the recognition result, and the black part represents the existence of musical instruments.Figure 8
Recognition effect of string music classification network in the second-level classification model.
(a)(b)As can be seen intuitively from the figure above, piano and violin can be accurately identified, while there is confusion in the viola, and the recognition accuracy needs to be improved.Figure9 is the identification effect diagram of the wind music classification network. As can be seen from the picture, the trumpet can be identified accurately, while there are confusions in the the other two instruments, and the recognition accuracy needs to be improved.Figure 9
Identification effect of the wind music classification network in the second-level classification model.
(a)(b)Figure10 is the identification effect diagram of the percussion music classification network. It can be clearly seen from the figure that all three musical instruments have been accurately identified, which means that there are obvious differences between these three musical instruments, so that they can be well identified and classified.Figure 10
Recognition effect of the percussion classification network in the second-level classification model.
(a)(b)The comprehensive analysis shows that the two-level classification model constructed in this article has the best comprehensive performance, and it has the more accurate recognition effect.To further verify the effectiveness of the proposed method, the experiment compares the accuracy of the proposed model with that of the existing duets, trios, and quartet, and the obtained comparison results are shown in Table7.Table 7
Comparison between the proposed model and existing methods.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyMA0.9120.9030.8490.9080.8550.8590.8090.7430.7620.8890.825Bipolar classification model0.9130.9010.8530.9070.8630.8660.8220.8110.8340.8800.857Duet0.8270.881—0.829——————0.841Trio0.6830.825—0.828——————0.778Quartet0.5730.799—0.791——————0.731As can be seen from the above table, compared with the other three methods, the recognition accuracy of the two-level classification model proposed in this article is as high as 85.7%. The recognition accuracy of duet is 84.1%. There are 77.8% for trio, and there are 73.1% for quartet. The method proposed in this article is much higher than the other three methods, which shows that the method proposed in this article has a higher recognition accuracy and better performance.


## 3.4.1. Multi-Instrument Recognition Results under Benchmark Model

There are ten kinds of musical instruments, which are divided into three categories: percussion instruments, string instruments, and wind instruments. The percussion instruments are xylophone, timpani, trumpet, and the string instruments are guitar, piano, bass, viola, and violin. In addition, the wind instruments are bassoon and saxophone. In the experiment, an appropriate experimental environment should be configured first, which is basically consistent with the previous pitch feature extraction experiment. The dataset is divided into two parts, namely, training set and test set. The ratio of the two parts is 9 : 1. Moreover, the possibility of the inexistence and unlabeled instruments in the training set should be considered.The extracted constantQ transform matrix XCQ88×165 is processed, which is spliced with S1–S6. The corresponding input matrices can be obtained, which can be represented as I1–I6, namely, the harmonic mapping matrix. Then, they are input into the model, and the corresponding class-time series matrix can be obtained. The feature changes in the benchmark model are shown in Table 1.Table 1
Change process of the feature size in the benchmark model.
Input sizeOperationOutput size176 × 165 × 12 × 1 Convolution kernel, 352 channels352 × 164 × 1352 × 164 × 13 × 1 Maximum pooling352 × 54 × 1352 × 54 × 13 × 1 Convolution kernel, 704 channels704 × 52 × 1704 × 52 × 13 × 1 Channels704 × 17 × 1704 × 17 × 12 × 1 Channels, 704 channels704 × 8 × 1704 × 8 × 11 × 1 Channels, 11 channels11 × 8The experimental results are obtained according to Table2. The instrument-type recognition results are evaluated through F1. In the total number of instrument recognition, there are three cases of unrecognized, misrecognized, and correct recognition. It can be seen that the overall accuracy can identify the proportion of accuracy times.Table 2
F1 and overall accuracy of ten instruments under the benchmark model (using estimated pitch).
Harmonic mapping matrix orderPianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyI10.880.880.840.880.820.820.760.730.770.800.77I20.890.890.840.890.840.830.780.750.730.810.77I30.900.90838.000 8920.830 8360.780.760.800.740.77I40.900.40832.000.890.840.840.790.750.750.790.77I50.890.890.840.890.850.850.800.770.800.760.77I60.890.890.830.890.850.850.040.770.770.790.78The real pitch label matrix is adopted to design the harmonic mapping matrixIng (n is 1–6). At this time, the values in Y1 are all accurate. According to the information in Tables 2 and 3, it can be clearly seen that compared with the estimated pitch labels, the harmonic mapping matrix obtained by using real labels can achieve higher overall accuracy and F1 value. In addition, compared with xylophone and timpani, the recognition results of estimated and real pitch of a small drum are basically the same, which is mainly related to the unfixed pitch. In this study, the pitch features are mainly used to recognize the musical instruments. So the recognition scores of different types of musical instruments are different. Compared with percussion instruments, the recognition scores of orchestral instruments are higher, which verifies the effectiveness of pitch feature extraction.Table 3
F1 scores and overall accuracy of ten instruments under the benchmark model (using real pitch).
Harmonic mapping matrix orderPianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyI1g0.900.900.850.890.840.830.770.740.770.830.80I2g0.910.910.860.890.850.830.800.760.750.850.80I3g0.910.910.850.910.850.840.790.770.760.840.80I4g0.910.910.850.900.850.860.810.780.770.850.81I5g0.910.910.840.900.860.870.820.790.800.810.80I6g0.910.910.850.900.860.870.770.780.790.820.80


## 3.4.2. Multi-Instrument Classification Results under Two-Level Recognition

Firstly, the configuration of the experimental environment is consistent with the previous section. The momentum algorithm (0.9) is adopted, where the weight attenuation factor, learning rate, and batch are 2 ∗ 10−4, 0.05, and 60, respectively. In this experiment, a tensor is input, including I1–I6. Furthermore, the output result is class-time series matrix. The specific characteristic changes are shown in Table 4.Table 4
Changes of feature size in the classification model based on the attention network.
Input sizeOperationOutput size176 × 165 × 62 × 1 Convolution kernel, 352 channels352 × 164 × 6352 × 164 × 63 × 1 Maximum pooling352 × 54 × 6352 × 52 × 63 × 1 Convolution kernel, 704 channels704 × 52 × 6704 × 52 × 63 × 1 Channels704 × 17 × 6704 × 17 × 62 × 1 Channels, 11 channels11 × 8 × 6704 × 17 × 6Attention subnetSix attention weights11 × 8 × 6The sum using the weighting of attention weight11 × 8I3 and I5 as the benchmark model of input are represented as BI3 and BI5, respectively. The classification model is represented as MA. According to the information in Table 5, it can be seen that compared with the previously adopted benchmark model, a higher overall accuracy is achieved by adopting the attention network model, and the recognition scores of all instruments except xylophone are improved.Table 5
F1 and overall accuracy and comparison of ten musical instruments based on the attention network classification model.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyBI30.900.900.840.890.830.840.780.760.810.730.77BI50.890.890.840.890.850.850.800.770.760.880.77MA0.910.900.850.910.860.860.810.740.800.860.83Based on the analysis of the above phenomenon, it is found that the attention network actually is to set the appropriate weights for the intermediate feature graph. Compared with the other types of musical instruments, the melodic instrument features are more conducive to recognition, which means the weights are higher. The common melodic instruments are piano and guitar, and the xylophone is rarely used as a melodic instrument. Therefore, if it exists at the same time with other instruments, the characteristics that need to be recognized can be easily masked. In this perspective, after adding the attention network, it is beneficial for melody instrument recognition, which is helpful to improve the overall accuracy. However, it is not possible to improve all instrument recognition scores, which needs to be further studied.In this article, the constantQ transform matrix of the first-level classification model is used as input and output instrument family-time series matrix. In the second-level classification model, the third-order harmonic mapping matrix I3, the fifth-order harmonic mapping matrix I5, and the sixth-order harmonic mapping matrix I6 are used as the input features of the string music classification network, wind music classification network, and percussion music classification network, respectively. Then, the output of the three networks is summarized to obtain the final instrument class-time series matrix. The recognition scores and overall accuracy of various musical instruments in the two-level classification model (MT) are obtained, and the comparison is shown in Table 6.Table 6
Comparison of F1 and overall accuracy of ten musical instruments in the two-level classification model.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyBI30.900.900.840.890.830.840.780.760.790.750.77BI50.890.890.840.890.850.850.800.770.710.830.77MA0.910.900.850.910.860.860.810.740.750.910.83MT0.910.900.850.910.860.870.820.810.830.890.86After comprehensively analyzing the charts, what can be found is that the recognition scores of most musical instruments are improved, especially xylophone. And it can be seen that the two-level classification model proposed in this article can balance the classification of musical instruments well, and the overall accuracy is further improved.Figure8 is the recognition effect diagram of the string music classification network. The upper part (a) represents the real label, the lower part (b) represents the recognition result, and the black part represents the existence of musical instruments.Figure 8
Recognition effect of string music classification network in the second-level classification model.
(a)(b)As can be seen intuitively from the figure above, piano and violin can be accurately identified, while there is confusion in the viola, and the recognition accuracy needs to be improved.Figure9 is the identification effect diagram of the wind music classification network. As can be seen from the picture, the trumpet can be identified accurately, while there are confusions in the the other two instruments, and the recognition accuracy needs to be improved.Figure 9
Identification effect of the wind music classification network in the second-level classification model.
(a)(b)Figure10 is the identification effect diagram of the percussion music classification network. It can be clearly seen from the figure that all three musical instruments have been accurately identified, which means that there are obvious differences between these three musical instruments, so that they can be well identified and classified.Figure 10
Recognition effect of the percussion classification network in the second-level classification model.
(a)(b)The comprehensive analysis shows that the two-level classification model constructed in this article has the best comprehensive performance, and it has the more accurate recognition effect.To further verify the effectiveness of the proposed method, the experiment compares the accuracy of the proposed model with that of the existing duets, trios, and quartet, and the obtained comparison results are shown in Table7.Table 7
Comparison between the proposed model and existing methods.
PianoViolinViolaGuitarSaxophoneBassoon tubeTimpaniXylophoneBassTrumpetOverall accuracyMA0.9120.9030.8490.9080.8550.8590.8090.7430.7620.8890.825Bipolar classification model0.9130.9010.8530.9070.8630.8660.8220.8110.8340.8800.857Duet0.8270.881—0.829——————0.841Trio0.6830.825—0.828——————0.778Quartet0.5730.799—0.791——————0.731As can be seen from the above table, compared with the other three methods, the recognition accuracy of the two-level classification model proposed in this article is as high as 85.7%. The recognition accuracy of duet is 84.1%. There are 77.8% for trio, and there are 73.1% for quartet. The method proposed in this article is much higher than the other three methods, which shows that the method proposed in this article has a higher recognition accuracy and better performance.


## 4. Conclusion

In conclusion, the two-level classification model based on the convolutional neural network proposed in this article has a good classification effect and recognition accuracy. It has certain validity. Through comparative experiments, it is found that the recognition accuracy of the proposed method is 1.6%, 8.1%, and 13.4% higher than that of the method of duet, trio, and quartet. So the recognition accuracy and classification effect of the proposed method are better. The validity of the proposed classification model is further verified by comparing the benchmark classification model with the classification model based on the attention network. However, due to the lack of experience and adequate experimental conditions, the research needs to be further improved and perfected. Specifically, the original audio and scores of various musical instruments can be added to obtain more datasets, so as to further improve the experiment scientificity.



---

*Source: 5117546-2022-03-25.xml*