# Beyond Spectrograms: Rethinking Audio Classification from EnCodec's Latent Space
## 3 Experiments
### Discussion  
In our experiments, the accuracy and loss values become worse as the number of classes increases. These results are due to the model having to learn more classes, making the task too complex for the simple network architecture used in this work. The results obtained with EnCodec's latent representation are still better than those obtained with the mel spectrogram. The worst results are obtained with the environmental sound dataset. However, the accuracy is well above 50% in both versions of the EnCodec model. It is worth noting that said dataset contains 50 classes. Leaving aside the comparison with the mel spectrogram, the model trained with EnCodec's latent representation obtains results comparable with those of the state-of-the-art audio classification using the same datasets, as shown in Table 2.  
To mention a few representative examples, ref. [17] obtains an accuracy of 99.70% in music and speech classification, 86.30% in music genre classification, and 94.70% in environmental sound classification using pre-trained neural networks. Similar results are achieved by [18] with masked autoencoders: 99.20% in music and speech classification, 86.10% in music genre classification, and 85.60% in environmental sound classification. Finally, using a technique based on audio transformers [19] obtains an accuracy of 96.80% in environmental sound classification. In [PERSON] et al. [19], the authors use mono audio with a sampling rate of 32 kHz. They extract mel features using a 1280 frames window with a hop length of 3200 frames, resulting in 128 mel bins. Similarly, [PERSON] et al. [18] preprocesses samples into log-scaled mel spectrograms with a sampling rate of 16,000 Hz, a window size of 3200 frames, a hop size of 3200 frames, and 80 mel-spaced frequency bins in the range of 50-8000 Hz. In contrast, [PERSON] et al. [17] resample all audio recordings to 32 kHz, convert them to monophonic and then apply STFT to the waveforms using a Hamming window of size 1024 and a hop size of 320 samples, resulting in 100 frames per second. They then apply 64 mel bins to compute the log mel spectrogram, with lower and upper cut-off frequencies set to 50 Hz and 14 kHz, respectively, to reduce low-frequency noise and aliasing effects. Our approach with mel spectrograms follows similar steps. We use the native sampling rate of the dataset (22.05 kHz for the GTZAN datasets and 44.1 kHz for ESC-50), 128 mel \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **Music/Speech** & \multicolumn{2}{c}{**Genres**} & \multicolumn{2}{c}{**Environmental**} \\ \hline  & **Accuracy** & **Loss** & **Accuracy** & **Loss** & **Accuracy** & **Loss** \\ \hline
**Mel** & 94.66\% & 0.47 & 69.50\% & 1.96 & 31.05\% & 3.90 \\
**24 kHz** & 96.09\% & 0.22 & 78.89\% & **0.87** & **58.85\%** & **1.53** \\
**48 kHz** & **96.88\%** & **0.17** & **80.84\%** & 0.90 & 54.25\% & 1.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Inference results (best values in bold). \begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Music/Speech** & **Genres** & **Environmental** \\ \hline
**EnCodec-based (24 kHz)** & 96.09\% & 78.89\% & 58.85\% \\
**EnCodec-based (48 kHz)** & 96.99\% & 80.84\% & 54.25\% \\ \hline
**[PERSON] et al. [17]** & **99.70\%** & **86.30\%** & 94.70\% \\
**[PERSON] et al. [18]** & 99.20\% & 86.10\% & 85.60\% \\
**[PERSON] et al. [19]** & - & - & **96.80\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Accuracy comparison with state-of-the-art models (best values in bold). bins, a window size of 512, and a hop size of 128 samples. However, to obtain the EnCodec feature map, the audio is resampled to match the model's required sampling rate, and then we apply the EnCodec compression. Although the parameters used in the experiments we compare against in Table 2 are not identical--since the models themselves are fundamentally different--the datasets remain the same. This consistency allows for a meaningful comparison of the results obtained by these models and our proposed approach.