# Beyond Spectrograms: Rethinking Audio Classification from EnCodec's Latent Space
## 2 Methodology  
EnCodec's architecture contains several components: an encoder, a quantizer, and a decoder. The encoder part of EnCodec is based on SEANet [11] and takes a raw audio sample as input. It converts it into a compressed latent representation using convolutional layers and a Long Short-Term Memory (LSTM) network through a series of steps. The audio signal first passes through a 1D convolution layer with 32 channels and a kernel size of 7, extracting the audio's initial features. After the initial convolution, the output is passed through four convolution blocks, each consisting of a residual unit and a downsampling layer. The residual unit helps to learn complex relationships between the features. At the same time, the downsampling layer reduces the dimensionality of the latent space. After the convolution blocks, a two-layer LSTM network is used to model the temporal dependencies in the latent representation to capture the time-varying nature of the audio signal. Finally, a 1D convolution layer with a kernel of size seven and a variable number of output channels is applied. The number of output channels depends on the target bit rate for the compressed audio. The encoder's output is a latent representation of the audio signal that is smaller in size than the original audio. This latent representation is then quantized and further compressed using entropy coding before being transmitted or stored. The quantizer then processes the encoder output. Its objective is to reduce the bit rate of the compressed audio signal while minimizing the loss of perceptual quality using RVQ [6], reducing the dimensionality of the latent representation. Using the quantizer output for audio classification could be an exciting approach. It could be easier and faster to process as it is smaller than the raw audio signal. However, more than this output is required to classify the audio signal, as there is no direct relationship between the quantized values and the audio signal's content. A 1-s mel spectrogram representation from a sample speech audio (Ashiel Mystery, A Detective Story, Ch. 2: [[https://librosa.org/doc/main/recordings.html#description-of-examples](https://librosa.org/doc/main/recordings.html#description-of-examples)]([https://librosa.org/doc/main/recordings.html#description-of-examples](https://librosa.org/doc/main/recordings.html#description-of-examples)), accessed on 1 February 2025) is shown in Figure 2a. The mel spectrogram is widely used in AI audio classification tasks. It is included as a layer in Keras 3.1([[https://keras.io/](https://keras.io/)]([https://keras.io/](https://keras.io/)), accessed on 1 February 2025) or as an output format in Kapre [12]. Unlike linear spectrograms, mel spectrograms use a frequency scale that mimics the human ear's sensitivity. They also outperform Short-time Fourier transform (STFT), as the resulting images are more compact. This makes them particularly effective in tasks involving audio analysis, such as speech and music processing. This approach has been widely adopted in the literature [13], demonstrating its effectiveness in audio classification tasks. For comparison, Figure 2b shows the encoder's output after processing the same sample but before the quantization, obtained from EnCodec's 48 kHz model. A naked-eye inspection of the visual patterns of EnCodec latent representation compared to the mel spectrogram clearly shows a reorganization of the data. The output of the EnCodec's encoder reduces the size of the audio and reorganizes it to facilitate its subsequent compression. We hypothesize that such reorganization may also be useful for audio classification, even though the encoder's output appears to be noise if played back as an audio signal. The nature of the audio signal is still present in EnCodec's encoder output in a way that is not directly perceptible to the human ear but that a neural network could learn to recognize.