# Beyond Spectrograms: Rethinking Audio Classification from EnCodec's Latent Space
## 1 Introduction  
The content we consume today, both for our education and leisure, has undergone a radical change in recent years, becoming mostly multimedia. Within this content, audio is one of the most important sources, whether in music, podcasts, audiobooks, or as part of an audiovisual product. Audio compression and classification are two essential tasks in the audio processing pipeline, as shown in Figure 1.  
Audio compression is crucial, enabling efficient audio content storage, transmission, and playback. By reducing the size of audio files, compression allows for significant storage savings. This is particularly important in an era where vast amounts of audio content are consumed and stored. Furthermore, compression facilitates faster and more reliable audio transmission over the Internet and wireless networks, as compressed audio files require less bandwidth, reducing download times. Due to its widespread compatibility, relatively high compression ratio, and acceptable audio quality for most listeners, MP3 has become a de facto standard for music distribution and download [1]. Streaming services like Apple Music and Spotify use AAC. While AAC offers superior audio quality comparedto MP3 at similar bit rates, MP3 remains the most popular audio format for distribution and download. Although MP3 is the de facto standard, its replacement by another format offering significant advantages is not unreasonable. This transition would be transparent to the user, considering its use is usually made through an intermediary, such as a web client, a music application, or a video player, to mention a few examples. Audio classification is used in various applications, such as categorizing audio content, enhancing music recommendation systems, automating metadata tagging, and aiding copyright detection in the music industry. It also plays a crucial role in music segmentation for production, facilitating research through trend analysis and improving sound engineering processes. The integration of technology and music optimizes industry workflows, enriches the user experience, and protects intellectual property. The scientific community has addressed the task of extracting the different features that an audio track may have for decades. Audio classification techniques were traditionally solved by extracting and processing audio features using probabilistic analysis. Some libraries, such as pyAudio-Analysis [2], Essentia [3], or librosa [4], allow audio feature extraction. However, adopting Artificial Intelligence (AI) techniques has revolutionized audio classification, allowing the task to be performed without requiring manual feature extraction. This made pyAudioAnalysis one of the first libraries to incorporate AI techniques for audio classification. Spectrogram-related features are the most widely used in AI audio classification tasks, as they provide a good representation of the audio signal's frequency content over time. A new audio compression algorithm, named EnCodec, has been recently proposed [5]. EnCodec is a neural audio codec that uses a streaming encoder-decoder architecture with quantized latent space to achieve high-fidelity audio compression in real-time. It provides up to 10 times less bandwidth than MP3 while maintaining the same audio quality. However, EnCodec is not the only neural audio codec of its kind, as this technology has gained significant research interest among major tech companies. Google's SoundStream [6] operates similarly to EnCodec, utilizing an encoder-decoder architecture that produces a latent representation of audio samples, which is then quantized using Residual Vector Quantization (RVQ). Nevertheless, EnCodec achieves a higher Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) ([[https://www.itu.int/rec/R-REC-BS.1534/en](https://www.itu.int/rec/R-REC-BS.1534/en)]([https://www.itu.int/rec/R-REC-BS.1534/en](https://www.itu.int/rec/R-REC-BS.1534/en)), accessed on 1 February 2025) score (53.1 vs. 67 for 3 kbit/s and 66.2 vs. 83.1 for 6 kbit/s).