# Beyond Spectrograms: Rethinking Audio Classification from EnCodec's Latent Space
## 2 Methodology
To test this paper's hypothesis, we trained an EnCodec-based audio classifier built on a neural network architecture ([[https://keras.io/examples/vision/mnist_convnet/](https://keras.io/examples/vision/mnist_convnet/)]([https://keras.io/examples/vision/mnist_convnet/](https://keras.io/examples/vision/mnist_convnet/)), accessed on 1 February 2025) based on the Modified National Institute of Standards and Technology (MNIST) ([[https://web.archive.org/web/20250112235228/http://yann.lecun.com/exdb/mnist/](https://web.archive.org/web/20250112235228/http://yann.lecun.com/exdb/mnist/)]([https://web.archive.org/web/20250112235228/http://yann.lecun.com/exdb/mnist/](https://web.archive.org/web/20250112235228/http://yann.lecun.com/exdb/mnist/)), accessed on 1 February 2025) classifier. The diagram in Figure 1 illustrates the relationship between the classifier and the entire system. The input layer is an array of \(128\times X\) obtained after preprocessing the audio samples, passing through convolutional and pooling layers. Input layer size, 128, is chosen to match the output size of EnCodec's encoder or the number of mel bands chosen to compute the mel spectrogram. \(X\) depends on the length of the computed frame and the preprocessing step (e.g., 150 items for 1 s of stereo 48 kHz EnCodec compression or 75 items for 1 s of mono 24 kHz) EnCodec compression. It is also variable depending on the number of samples between successive frames when computing the mel spectrogram (e.g., 173 when using 256 as the hop length for a 44.1 kHz audio). Once the audio file is loaded, it is represented as a vector of length \(N\times SR\), where \(N\) is the duration of the audio in seconds and \(SR\) is the sampling rate. In the case of multi-channel audio, such as stereo files, the representation takes the form of a matrix with dimensions (\(NC\), \(N\times SR\)), where \(NC\) is the number of channels and \(N\times SR\) corresponds to the number of samples per channel. For the calculation of the mel spectrogram, the first step in our process is to convert the multi-channel audio to mono format. This is achieved by averaging the values across  
Figure 2: One second sample audio signal comparison. all channels, thereby reducing dimensionality and facilitating subsequent processing. Once converted to mono, the audio is segmented into fragments of \(WINDOW\) seconds using a sliding window that advances \(STEP\) seconds between each segment. This approach generates a series of uniform-sized segments that are used for model training and prediction. The result of this process is a set of segments represented as a vector with dimensions (\(M\), \(WINDOW\times SR\)), where \(M\) is the total number of segments generated. Each segment corresponds to a specific fragment of the audio that is processed individually by the neural network. This methodology enables not only the classification of the entire audio file but also its precise segmentation, which is particularly useful for analyzing changes within a single track. The process of audio segmentation using a sliding window can be described as follows. In this example, an audio file with a duration of 10 s is used, and a sliding window with a length of \(WINDOW=2\) s and a step of \(STEP=1\) s is applied. Since the window size is twice the step size, there is overlap between adjacent fragments, which is advantageous for preventing the loss of critical information. The first window fragment will capture the audio interval \([0,2]\), while the second will capture the range \([1,3]\), and so on. This overlap ensures that audio elements near the edges of each window are not left unprocessed, which is crucial in situations where changes in the audio occur gradually. For the final window segment, padding is applied using NumPy functions. Instead of assigning silence values, this padding replicates the audio content in an inverted manner, which helps to prevent potential bias in the model.