## 3 Denoising Transformer-Based Audio Music Genre Classification
### Pre-Training Stage  
The objective of the pre-training stage is to enable [PERSON] to understand the deep representation of audio music through unsupervised denoising. The role of the prior decoder within this framework is to restore patches from low-dimensional hidden states obtained from [PERSON] during pre-training. An autoencoder (AE), which comprises an encoder and a decoder, was designed to train the decoder, as shown in Figure 4. The encoder consists of max-pooling and convolutional layers, and the decoder consists of convolutional and up-sampling layers. The encoder compresses the input patches into low-dimensional vectors, and the decoder restores the low-dimensional hidden states to the original patches. The mean squared error (MSE) loss is calculated to update the AE parameters.  
To complete the denoising, patches of noisy Mel spectrograms \(\{p_{1}{}^{\prime},p_{2}{}^{\prime},p_{3}{}^{\prime},...,p_{n}{}^{\prime}, \ldots p_{N}{}^{\prime}\}\) are passed into Deformer, as shown in Figure 5. The position embedding layer, which is trainable, utilizes absolute numerical embedding to integrate positional information into these patches. The transformer layers utilize multi-head self-attention and feed-forward neural networks to relate to this layer. Subsequently, the prior decoder restores these low-dimensional hidden states back into the restored patches \(\{p_{1}{}^{*},p_{2}{}^{*},p_{3}{}^{*},...,p_{n}{}^{*},\ldots p_{N}{}^{*}\}\).  
Figure 4: Structure of the autoencoder.  
Figure 5: Pre-training stage of [PERSON].  
Then, the MSE loss is calculated based on the restored patches \(\{{p_{1}}^{*},{p_{2}}^{*},{p_{3}}^{*},...,{p_{n}}^{*},\ldots,{p_{N}}^{*}\}\) and original patches \(\{p_{1},p_{2},p_{3},...,p_{n},\ldots,p_{N}\}\) for training [PERSON]. This training strategy allows [PERSON] to gain a deep understanding of the contextual relationships and interdependencies among patches.  
Algorithm 1 describes the pre-training stage, where Deformer(\(\bullet\)) represents Deformer and \(\theta\) represents Deformer parameters. \(I\) denotes the number of training steps. \({p_{n}}^{\prime}\) represents one of the patches from the noisy Mel spectrograms, and \({p_{n}}^{*}\) is the restored patch. MSE(\(\bullet\)) represents the loss function to calculate the loss \(L\) between \({p_{n}}^{*}\) and \({p_{n}}\), where \({p_{n}}^{*}\) is only generated by the injected noise \(z\) and \({p_{n}}\) is the patch of the normalized Mel spectrogram without noise. \(\theta\) is updated based on a gradient, which is calculated as \(-\eta^{*}\
abla L(\theta)\), where \(\eta\) is the learning rate.  
```
Input: \(\{p_{1}^{\prime},p_{2}^{\prime},p_{3}^{\prime},...,p_{n}^{\prime},\ldots p_{N }^{\prime}\}\) Output: \(\{p_{1}^{*},p_{2}^{*},p_{3}^{*},...,p_{n}^{*},\ldots p_{N}^{*}\}\)
1: Initialize Deformer (\(\theta\))
2: for \(i=1\) to \(I\) do:
3: Forward pass: \({p_{n}}^{*}=\) Deformer (\({p_{n}}^{\prime},\theta\))
4: If \({p_{n}}^{\prime}={p_{n}}+z\) :
5: Compute \(L=\) MSE(\({p_{n}}^{*},{p_{n}}\))
6: Update \(\theta=\theta-\eta^{*}\
abla L(\theta)\)
7: End for
```  
**Algorithm 1** Pre-training