Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Authors: Dianwen Ng
Kun Zhou
Yi-Wen Chao
Zhiwei Xiong
Bin Ma
EngSiong Chng
Venue: ICML 2025 poster (2025)
Keywords: Neural Audio Codec
Neural Psychoacoustic Codec
Zero-shot TTS
Vector Quantization
Primary area: deep_learning->other_representation_learning

Abstract: Achieving high-fidelity audio compression while preserving perceptual quality across diverse audio types remains a significant challenge in Neural Audio Coding (NAC). This paper introduces MUFFIN, a fully convolutional NAC framework that leverages psychoacoustically guided multi-band frequency reconstruction. Central to MUFFIN is the Multi-Band Spectral Residual Vector Quantization (MBS-RVQ) mechanism, which quantizes latent speech across different frequency bands. This approach optimizes bitrate allocation and enhances fidelity based on psychoacoustic studies, achieving efficient compression with unique perceptual features that separate content from speaker attributes through distinct codebooks. MUFFIN integrates a transformer-inspired convolutional architecture with proposed modified snake activation functions to capture fine frequency details with greater precision. Extensive evaluations on diverse datasets (LibriTTS, IEMOCAP, GTZAN, BBC) demonstrate MUFFIN’s ability to consistently surpass existing performance in audio reconstruction across various domains. Notably, a high-compression variant achieves an impressive SOTA 12.5 kHz rate while preserving reconstruction quality. Furthermore, MUFFIN excels in downstream generative tasks, demonstrating its potential as a robust token representation for integration with large language models. These results establish MUFFIN as a groundbreaking advancement in NAC and as the first neural psychoacoustic coding system. Speech demos and codes are available at \url{https://demos46.github.io/muffin/} and \url{https://github.com/dianwen-ng/MUFFIN}.

Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Dianwen Ng! 12 Kun Zhou
Yi-Wen Chao Zhiwei Xiong²
Bin Ma
Eng Siong Chng²
Abstract
1. Introduction
Achieving high-fidelity audio compression while preserving perceptual quality across diverse au- dio types remains significant challenge in Neu- ral Audio Coding (NAC). This paper introduces MUFFIN, fully convolutional Neural Psychoa- coustic Coding (NPC) framework that lever- ages psychoacoussicalll guided multi-band fre- quency reconstruction. Central to MUFFIN is the Multi-Band Spectral Residual Vector Quantiza- tion (MBS-RVQ) mechanism, which quantizes latent speech across different frequency bands. This approach optimizes bitrate allocation and enhances fidelity based on psychoacoustic stud- ies, achieving efficient compression with unique perceptual features that separate content from speaker attributes through distinct codebooks. MUFFIN integrates transformer-inspiee con- volutional architecture with proposed modified snake activation functions to capture fine fre- quency details with greater precision. Exten- sive evaluations on diverse datasets (LibriTTS, IEMOCAP, GTZAN, BBC) demonstrate MUF- FIN's ability to consistently surpass existing per- formance in audio reconstruction across various domains. Notably, a high-compression variant achieves an impressive SOTA 12.5 Hz rate while preserving reconstruction quality. Furthermore, MUFFIN excels in downstream generative tasks, demonstrating its potential robust token rep- resentation for integration with large language models. These results establish MUFFIN as a groundbreaking advancement in NAC and as the first NPC system. Speech demos and codes are available 2
Neural Audio Coding (NAC) has emerged as a transforma- tive technology in speech and audio processing, enabling high compression and high-fidelity reconstruction of audio signals with substantially reduced data representation sizes (Défossez et al., 2022; Du et al., 2024b; Wu et al., 2023; Kumar et al., 2024). This advancement not only optimizes storage and transmission efficiencies but also enhances the integration of semantic and acoustic details into speech- large language models (LL.Ms) (Zhang et al., 2023a; 2024; Défossez et al., 2024). Within NAC, neural autoencoders and quantization techniques have achieved significant suc- cess in building unit representations that effectively capture essential audio features and compress them into precise, compact tokenized forms. Among these quantization meth- ods, Residual Vector Quantization (RVQ) has emerged as the choice of method to enhance information preservation by refining the quantization process through multiple boosting turss (Zeghidour et al., 2022). This progressive refinement reduces reconstruction errors, enhancing fidelity to the orig- inal signal, ensuring higher quality audio outputs.
Psychoacoustiess (Liu et al., 2017; Zhen et al., 2020; Byun et al., 2022). the study of how humans perceive sound, under- pins traditional audio coding frameworks such as MP3 and OPUS (Herre & Dick, 2019), yet remains largely unexplored in NAC. This discipline offers crucial insights for designing perceptually oriented systems, particularly through princi- ples like perceptual masking. Different frequency bands carry distinct types of information: low frequencies are critical for speech intelligibility, mid frequencies capture formant structures essential for content articulation, and high frequencies convey speaker identity, pitch, and tim- ire-aatributes integral to naturalness and spatial realism (San Roman et al., 2024; Petermann et al., 2023). By seg- menting and encoding these bands separately. we account only for perceptually relevant distortions, reducing effective entropy while allowing greater controllability of learned speech attributes. This psychoacoustic perspective has the potential to redefine NAC, guiding more efficient compres- sion strategies and richer, more robust reconstructions.
'MiroMind, Singapore *College of Computing & Data Science, Nanyang Technological University. Singapore 'Tongyi Speech Lab, Alibaba Group Singapore. Correspondence to: Dianwen Ng <dianwen001 @ edu....d..sg>
Proceedings of the rhnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Leveraging psychoacoustic insights, we introduce MUFFIN, the first neural psychoacoustic codec (NPC) that utilizes the proposed multi-band spectral RVQ to achieve high-fidelity
'https: //ee:///dmmm//// 6, git..b............. fo/mmffin/////// Pttpps: / thub. commcom////oooo www.nnnnnng////ememeee /00mm MMFFIN

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
audio compression. MUFFIN aligns compression strategies with psychoacoustic principles, achieving an optimal bal- ance between bitrate efficiency and perceptual quality, Ad- ditionally, we have enhanced frequency modeling through a novel modification of the snake activation function. Ex- tensive experimental evaluations demonstrate MUFFIN's superior performance in audio reconstruction compared to existing NACs. Furthermore, our codec excels in important downstream tasks, such zero-shot text-to-speech (TTS), showcasing its potential as robust token representations for integrating with LLMs for more advanced applications.
neural audio coding better address the complexities of auditory perception.
Multi-band Audio Processing Human auditory perception operates through multi-band system, where the cochlea acts as frequency analyzer, separating complex waveforms into bands that deliver crucial perceptual cues (Pulkki & Karjalainen, 2015). Psychoacoustic principles like critical bands and frequency masking are foundational to models such the Bark scale (Zwicker & Terhardt, 1980) and Equivalent Rectangular Bandwidth (ERB) (Moore & Glas- berg, 1983), reflecting human auditory sensitivity to fre- quency variations. Leveraging these principles, multi-band modeling has significantly advanced audio compression by enhancing perceptual efficiency (Zhen et al., 2020).
Contributions (1) We propose a novel multi-band spectral RVQ to optimize bitrate allocation, enhancing compression efficiency and perceptual audio quality. (2) We introduce MUFFIN, the first neural psychoacoustic codec that incor- porates modified snake activation function to achieve su- perior audio reconstruction at a SOTA 12.5 Hz compression while preserving high fidelity. (3) Our empirical demonstra- tions and experiments analyze the characteristics of each novel perceptual codebook, offering a better understanding of their potential use in various audio processing tasks.
Legacy codecs like MPEG-1 Audio Layer III (MP3) and MPEG-2 Advanced Audio Coding (AAC) (Bosi et al., 1997) used these psychoacoustic models to dynamically allocate bits based on perceptual thresholds, achieving near- transparent sound quality. However, their static compression thresholds limited adaptability to complex audio signals. Modern neural audio codecs (Xiao et al., 2023; Luo et al., 2024; Langman al., 2024; Nishimura al., 2024; Chen et al., 2024) have overcome these limitations through flex- ible neural architectures that adapt more dynamically to signal complexities. For instance, the Penguins codec (Xiao et al., 2023) enhances perceptual audio quality by segment- ing audio into frequency bands, using generative models for low frequencies and bandwidth extension for high fre- quencies to reduce bitrate without compromising quality, By quantizing mel-band features, spectral codecs (Langman et al., 2024) align more closely with human auditory percep- tion, although this can sometimes impact the accuracy of spectral details and phase information. Additionally, Gull (Luo et al., 2024) uses the Band-Split RNN (BSRNN) (Luo & Yu, 2023) architecture to effectively capture temporal and inter-band dependencies through subband modeling and neural compression. However, its stacked BSRNN blocks and causal RNN layers increase computational latency, im- pacting real-time streaming applications.
2. Related Work
Neural Audio Codec Neural audio compression frame- works such as Soundstream (Zeghidour et al., 2022) and EnCodec (Défossez et al., 2022) have advanced the field by implementing fully convolutional encoder-decoder architec- tures with RVQ bottlenecks, optimizing lossy tokenization while maintaining high fidelity. These models use HiFi- GAN vocoder losses (Kong et al., 2020) to balance low bitrates with accurate reconstruction, ensuring robust audio quality even at reduced bandwidths (Ai et al., 2024). Au- dioDec (Wu et al., 2023) further enhances this approach by integrating grouped convolutions for real-time process- ing in streamable format, while HiFi-Codec (Yang et al., 2023) introduces parallel gvo layers minimize redundancy. However, challenges persist. RVQ-based mod- els often fail to preserve fine-grained details in complex acoustics, leading to perceptible artifacts (Langman et al., 2024). Transformer-based NAC models, like SpeechTok- enizer (Zhang et al., 2023b), which disentangle semantic content from the remaining audio features, face limitations in generalizing to input sequences longer than those en- countered during training (Variš& & Bojar. 2021; Chen al., 2023), resulting in inconsistent performance and diminished output fidelity. Moreover, the prevalent practice of applying quantization across full frequency bands overlooks potential efficiency gains that could be achieved by accounting for the perceptual importance of different spectral regions, which effective for achieving high compression capability while maintaining high-fidelity audio (Petermann et al., 2023). These issues highlight the need for continued innovation in
3. Methods: Multi-Band Frequency Coding
3.1. Multi-band spectral residual vector quantization (MBS-RVQ)
In traditional methods, a multi-band model divides the fre- quency domain of a discrete- time signal, 2 2/[n], into different frequency bands using the Fast Fourier Transform (FFT). Mathematically, the FFT of the signal can be expressed as:
X >.N' i k= = N N-1 (1)
where X[k) represents the complex frequency components of the signal, N is the total number of samples, and l is the
2

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
frequency index. We define K non-overlapping frequency bands, each corresponding to specific range of frequencies, denoted as B3 can be defined as:
each subband quantizer will be accounted for in the sub- sequent subband feature input as residuals, following the RVQ approach, as we leverage the principle of successive refinement. This method enables each stage to quantize the residual error from the previous stage, thereby enhancing the preservation of detail in the audio representation. Figure presents an illustration of the proposed MBS-RVQ.
B₁ {f: fmin, S f < ] k 2., K
where fuin,k and fmax.& denote the minimum and maxi- mum frequencies of the ktth band, respectively. After split- ting the signal into frequency bands, we are able to construct the multi-band neural tokenizer by feeding each band with a typical time-domain neural autoencoder. The spectral matrix of each band is converted back into a time-domain waveform using the inverse FFT. This waveform is then passed through the encoder for quantization. Although this approach is straightforward for processing the signal in sep- arate bands, it increases latency, as each band-pass signal requires individual encoding steps through the autoencoder (i.e., scaling the computational burden in terms of FLOPs by the number of bands) before being recombined to reconstruct the original signal. This added computational complexity limits its suitability for real-time streaming ap- plications.
Besides, adopting a multi-band splitting approach enhances the expression of individual unit codebooks within their re- spective spectral bands during the quantization process. This method is informed by psychoacoustic research, which high- lights how different frequency ranges convey distinct types of information (Appendix A). Specifically, low- uow-frquency bands are essential for speech intelligibility due to their concentrated energy, while mid-frequency bands are crucial for articulating content through formant structures. High- frequency bands provide detailed acoustic features such as speaker identity. pitch, and timbre, contributing to the natu- ralness of speech (San Roman et al., 2024; Petermann et al., 2023). Furthermore, these higher frequencies enhance the spatial and ambient qualities of audio recordings.
By applying this knowledge, the encoding process can be optimized to allocate resources more effectively across the audio spectrum, ensuring that each data segment is pro- cessed to maximize the preservation and clarity of key audi- tory cues. This strategy not only allows for the perceptually meaningful quantization of audio units but also improves the efficiency and effectiveness of data compression, thereby significantly improving the overall perceptual quality of the output at much lower codebook bitrate.
Sheneoo
Spectral Split (Fast Fourier Transform))
Water Gannren
Low conttaaaryy..
Figure 1. Illustration of the MBS-RVQ process: Fast Fourier Trans- form (FFT) is applied to the encoded latent representation to isolate specific frequency bands, capturing targeted spectral information for each codebook. The filtered representation is reconstructed us- ing inverse FFT hefore undergoing quantization. The quantization residuals are then passed to the next codebook
3.2. Analysis: Multi-band modeling improves generative quality by leveraging the perceptual entropy bound.
To provide justification for MUFFIN, let x(I) be continuous-time audio signal defined over t E R. and let xtt) be its compressed approximation. Suppose the hu- man auditory system is modeled by a set of perceptual fil- ters that divide the frequency axis into K critical bands B1, B2...., Bk. Each band B₁ corresponds to a region where the ear has distinct sensitivity levels (c.g.. Bark or Mel scales). For each band BR, let PA (x)+) represent the perceptual threshold function indicating the maximum al- lowable distortion before artifacts become noticeable, and let D& (x(t). (x)) denote the band-specific distortion in- troduced by compression. The perceptual entropy E, of x(1) is defined as the minimal bit rate, RE, needed so that (x(t), x(t) $ Ps (x(t)) for all k. ensuring transparent audio compression:
Therefore, rather than splitting the bands at the input, we propose operating the multi-band processing within the en- coder's latent space using its latent features. The audio is first encoded into compressed representation 2, capturing both spectral and temporal features. The latent representa- tion 2 E RdxT, where d is the channel dimensionality and T's is the temporal length, is then decomposed into multiple frequency bands. Spectral band splitting is performed using the FFT, as previously detailed. We specifically target fre- quency bands of 0 to 18.75 Hz, 18.75 to 37.5 Hz, and 37.5 to 75 Hz, with corresponding scale factors of 4, 2, and 1, respectively. These scale factors applied within the 75 Hz bandwidth of the latent representation, which is derived from compressing 24 kHz audio by a factor of 320 using the encoder. Each band is then quantized sequentially and continuously updated through an exponential moving aver- age (Défossez et al., 2022). Note that the residuals from
Ep Ek_ min( (x(x(t), x(1) S Px(x())))
Theorem 3.1 (Perceptual Entropy and Masking Bounds). (Cover, 1999) For the given audio signal x(t) and com-
3

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
pressed representation x()), the percepiual entropy Eg sat- isfies the following lower bound when optimal multiband modeling is employed:
Predict Spuctrogom 903)
Spectragare
Ercoder
Dacadar
Ep 2 E H(B& (x(t) - E x(),
MBS RVQ
Iepur
Resenetrccte
where H1 (Bx] x(t))
is the Shannon entropy of the signal components within band Bx- and (B,r x(t)) is the per- ceptual masking effect that reduces the effective entropy by accounting for inaudible distortions in band Bs-
Speech
Spench
XOO
Disortmvintoro
Rneoding/Decodiin /Decoding Black
Rea BorBlockkapee (CNN)
Theorem 3.1 characterizes how multiband modeling lever- ages psychoacoustic properties to achieve lower bit rates without sacrificing perceived audio quality. The first term, Ek (Bx x(t)), represents the total intrinsic entropy over the K critical bands, analogous to the information- theoretic bound one would calculate if there were no mask- ing effects. However, human hearing does not require per- feet fidelity in all frequency regions; many distortions re- main hidden under the masking threshold (Zwicker & Fastl, 2013). This phenomenon allows codecs to allocate fewer bits to masked regions without degrading the subjective audio experience.
con-
Modiiiee Seriion Activation
Cors 10 10 di
Modifled Activation Cona 10
Pealler
layers
Figure 2. Architecture of MUFFIN incorporating fully convolu- tional structure.
learn richer, more detailed representations, shown in Figure 2. To reduce latency by improving the computa- tional efficiency from the channel-upsampling layer, we employ grouped convolutions that upscale in groups of 32 channels, which significantly reduces the number of model parameters and computations. The model consists of 46.1M parameters, with 34.2M in the encoder and 1 11.9M in the decoder. The Multiply-Accumule reach 3 31.6G per second of audio sampled at 24 kHz, demon- strating enhanced performance efficiency compared to the HiFi-Coded model, which has 61.5M parameters and tak- ing 44.4G of MACs computational steps.
In mathematical terms, A(BE, x(t)) represents the "mask- ing offset" that effectively reduces the bits needed for band Bx. Due to this offset, the total perceptual entropy Eg can be substantially smaller than the naive entropy sum, reflect- ing how frequency regions with strong masking require fewer bits for transparent encoding. This principle underlies the design of many perceptual audio codecs (Brandenburg & Bosi, 1997), where an FFT first decomposes the signal into subbands aligned with the human ear's 's critical bands, and then quantization is adapted based on psychoacoustic models (Johnston, 1988).
Operations (MACs)
It is noteworthy that our convolutional block mirrors trans- former functionality. The communication block from MRF mimics self-attention, enabling global temporal interactions, while the complexity block from the inverted bottleneck layer, adding depth parallels the transformer's 's transformeriisppreedd architecture and expressiveness. This transformer efficiently captures hierarchical temporal patterns, ensuring robust audio signal reconstruction performance.
3.3. Model Architecture
MUFFIN features an autoencoder architecture inspired by HiFi-Codec (Yang et al., 2023), utilizing a fully convolu- tional encoder-decoder network for temporal downscaling. We adopt the same striding configuration (2, 4, 5, 8), opti- mized for 24kHz audio waveforms, achieving total down- sampling factor of 320 in the default configuration. A key component of our convolutional block is the multi-receptive field (MRF) fusion, adapted from Kong et al. (2020), which aggregates outputs from residual blocks with varying di- lated kernel sizes. This allows the model to capture de- pendencies across multiple temporal scales, enhancing its capacity handle long-range sequential information. To enhance the model's 's representational power, we integrate an inverted bottleneck layer with residual skip connections, inspired by ConvNeXt (Liu et al., 2022), which increases channel dimensions and adds neural complexity. This de- sign aligns with SOTA architectures (Dao & Gu, 2024; Yu & Wang, 2024; Han et al., 2024), enabling the model to
Periodic Activation Function. To enhance periodic model- ing for better spectral preservation and high-fidelity recon- struction, we draw inspiration from Kumar et al. (2024) by replacing all Leaky ReLU activations with the snake acti- vation function (Lee et al., 2022), x + 1/a sin' ax, which better preserves high-frequency information (Ziyin et al., 2020) and maintain Lipschitz continuity, since the derivative of snake activation function is bounded by a constant of 1. We further enhanced this function by introducing amplitude and bias adjustments to improve overall performance. In the original formulation, the parameter a controls the frequency of the periodic component, while its reciprocal 1/a, attenu- ates the amplitude. To overcome this limitation, Evans et al. (2024) introduced learnable scaling factor 3 to adjust the amplitude independently. However, this adjustment may
4

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
result in increased variance because of the amplified peri- odic magnitude relative the input (Appendix D, Figure 4). To address this, we propose adding bias term 7 that learns to shift the output, ensuring better adaptation to the new scale while preserving consistency with the input range. This approach mitigates variance issues and offers greater flexibility in fitting data, greatly enhancing high-fidelity reconstruction.
anechoic recordings of speech (585 and 100 hours, respec- tively). For music, utilize Music4All (Santana al., 2020) (910 hours). For environmental sounds, we use ESC- 50 (Piczak, 2015) (3 hours, 50 classes with 40 examples per class, loosely arranged into 5 major categories: ani- mal, human, natural sounds, interior, and exterior sounds). Music and environmental sounds are used MUFFIN to lear broader audio expression, enhancing the foundational representations. All audio was resampled to 24 kHz.
3.4. Training Objectives
Prior to training, all audio files are truncated to a maximum duration of 10 seconds. A text file containing the paths to the processed audio files is provided to the data loader. This ensures a balanced sample distribution of speech and mu- sie file samples, mitigating data imbalance and preventing under-performance skewing towards vocal or instrumental reconstruction. During training, we apply voice activity de- tection to remove non-audio segments, optimizing learning efficiency. For each batch, 1-second audio segments are randomly selected from each instance and are zero-padded if shorter than 1 second. No additional data augmentation is applied to maintain simplicity in the experiment.
Following the framework from HiFi-Codec, our training leverages two key components in the overall loss objectives.
Reconstruction Loss We employ multiscale mel spec- trogram reconstruction loss, calculated as the LI distance between predicted and target mel-spectrograms over multi- ple time scales (i.e., a 64-bins mel-spectrogram derived from an STFT, with a window size of 2 and a hop length of 22/ /4 for = 7, 9, 10, 11). Unlike Yang et al. (2023) that uses 80 mel-spectrogram bins, we opted for lower bin count based on perceptual evaluations, which revealed improved naturalness in the reconstructed audio while preserving key spectral features. Although reducing the bin count compro- mises frequency resolution, our results indicate that stricter distance reconstruction criteria, while preserving more in- formation, may slightly harm perceptual quality.
Model and training details. In our experiments, models were trained on two A800 GPUs for 300K iterations with learning rate of 2e-4 and batch size of 20 per GPU, All quantizers utilize 9-bit code lookup from the EMA codebook. Additionally, we developed a low token-rate compression variant of MUFFIN to leverage the model's s efficiency in capturing non- nod-radundant information across dif- ferent frequency bands. The highly compressed MUFFINs increase the downsampling rate by 960x and 1920 in the encoding layers, producing latent representations at 25 Hz and 12.5 Hz, respectively. This configuration achieves au- dio quantization at 150 and 100 tokens per second, utilizing additional residual codebooks with a total of six and eight vector quantization layers. The same MBS-RVQ configura- tion is used, maintaining a 4:2:1 frequency ratio relative to the sampled frequency of the latent embeddings (partitioned logarithmic scale) for the default band splits. MUFFIN, operating at 12.5 H7, is trained on 2-second audio segments and requires four A800 GPUs to support a batch size of 10 per GPU. Using a lower learning rate is crucial to prevent gradient explosion. The high-compression MUFFIN model comprises approximately 50.6M parameters, with 36.5M in the encoder and 14. in the decoder (both models share the same architectural depth, differing only in downsam- pling rates). It achieves significantly lower MACs at 17.9G per second of audio, enabling faster inference.
Discriminative Loss We use three discriminators: a multi- scale STFT discriminator (MS-STFT) (Zeghidour et al., 2022), a multi-period discriminator (MPD), and a multi- scale discriminator (MSD) (Kong et al., 2020) to enhance perceptual quality through adversarial learning. We adopt the HingeGAN (Lim & Ye, 2017) adversarial loss formula- tion and L1 feature matching loss (Kumar et al., 2019).
MBS-RVQ Comnitment Loss We adopt quantization with commitment losses based on the VQ-VAE framework (Van Den Oord et al., 2017),, using stop-gradients and the straight- through estimator (Bengio et al., 2013) for backpropaga- tion through the codebook lookup. The input is split into K critical frequency bands (B,), where B, denotes the i-th band. For each band, the encoder output (B.) (en- coded representation) is mapped to the nearest codebook vector zi (P.) (quantized representation), ensuring frequency- specific representation. The commitment loss is defined as Ccommit = Σ 3.112( (B.) 24 (B2)) enabling precise mapping over the quantization of each spectral band.
4. Experiments
Since our training setup closely adheres to Hifi-Codec, one of the leading codec model in the field, we establish as our baseline by retraining the model with the same configuration, allowing for accurate performance comparison. Addition- ally, we benchmarked our method against other prominent
Data sources. We train our model on a modest collec- tion of 1,600 hours of speech, music, and environmental sounds. For speech, we use LibriTTS (Zen et al., 2019) and EARS (Richter al., 2024) datasets with expressive
5

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
codecs, including OPUS, Encodec and DAC, all reconfig- ured to transmission rate of 3.0 kB/s to assess performance at similar transmission rate as our codec reconstruction model. Furthermore, we evaluate high-compression MUF- FIN variant against Mimi, the recent SOTA codec model for 12.5 Hz, to highlight the strengths of our work.
formly improve all objective metrics. In fact, as the number of codebooks increases, especially higher compression rates, some objective measures experience a decline. This decline may be attributed to the introduction of noise or artifacts by additional codebooks, which, while capturing more detail, also amplify aspects that negatively impact certain evaluation metries. Thus, the relationship between increased codebooks and system performance exemplifies trade-off between improved naturalness and the potential degradation of other audio quality metrics. Nevertheless, it is notable that MUFFIN A remains competitive with leading NAC models and even outperforms Mimi in terms of natural- ness and reconstruction fidelity, achieving significant gains in audio quality while maintaining efficient compression rates. Table 2 showcases MUFFIN's zero-shot reconstruc-
Evaluation objectives. We utilize the objective evaluation metrics outlined in codec-SUPERB (Wu et al., 2024) to assess the perceptual quality of different audio domains. Specifically, we incorporate metrics such as the Perceptual Evaluation of Speech Quality (PESQ) (Rix et al., 2001), Short-Time Objective Intelligibiliity (STOI) (Taal et al., 2010), STFT distance (Alsteris & Paliwal, 2007), Mel dis- tance (Kubichek, 1993), and FOCORR (FO Pearson Corre- lation Coefficient) (Jadoul al., 2018). The selection of these metries for the corresponding audio domain is justi- fied by the inclusion-exclusion criteria discussed in codec- SUPERB. Additionally, we employ two automated Mean Opinion Score (MOS) evaluation metrics, UTMOS (Saeki et al., 2022) and ViSQOL (Hines et al., 2015), to assess the perceptual quality of the codecs. These metrics are designed to closely approximate subjective listening tests, providing a more accurate and robust evaluation of codec performance. For speech evaluation, we use the test-clean and test-other set from LibriTTS and evaluate emotional speech reconstruction with IEMOCAP (Busso et al., 2008). For music, we employ the GTZAN dataset (Sturm, 2013), while environmental sounds are evaluated using audio from the BBC sound effects (LAION-AI, 2022).
Table 1. Objective evaluation of reconstructed speech from the LibriTTS dataset using various neural audio codce models. GT refers to the abbreviation for ground truth. Except for highly- compressed MUFFIN and Mimi, which uses a compression rate of V x960 (25.0 Hz) and x x920 (12.5 Hz), the others have the compression rate of x320 (75 Hz). Note that HiFi-Codec was retrained using the official configuration settings, but on the same dataset as MUFFIN.
Test-Clean (Libn'TT)
Model OPLS
STFT
MEL
PESQ
STOI
UTMOS
ViSQOL
4.041
5.728
2.796
1.132
0.715
1.264
2,888
Encodoc
1.956
1.051
2.042
0.903
2.269
4.078
DAC
1.759
0.849
2.370
0.915
2.951
4.143
HitiiCCode
1.618
0.765
2.712
0.943
3.831
4.410
Mimi
2.488
1.706
1.715
0.620
2.966
3.791
4.1. Experimental Results
MUFFIN
1.555
0.692
2.996
0.954
4.017
4.516
MUFFIN V
1.626
0.755
2.525
0.937
4.035
4,345
Table and 2 compare the speech reconstruction quality of our MUFFINS against existing top performing NACs. We indicate that NACs listed in the table operate at band- width of 3.0 kB/s, except for Mimi at 1.0 kB/s, MUFFIN (default) at 2.7 kB/s, MUFFIN (V) at 1 1.35 kB/s, and MUF- FIN (A) at 0.9 kB/s. We use LibriTTS evaluation dataset, with 4,837 samples for test-clean and 5,120 for test-other. Notably, MUFFIN achieves superior reconstruction fidelity, achieving the lowest distance error and outperforming com- peting NACs across various objective metrics. This includes HiFi-Codec, which serves as robust baseline by being retrained under the same conditions, thereby confirming the effectiveness of the proposed framework. Furthermore, MUFFIN V and A achieve better UTMOS scores despite reductions in bandwidth and token rates. It appears that increasing the number of codebooks in our NAC system enhances the preservation of information from the original speech, resulting in improved perceptual quality as reflected by higher UTMOS scores. This suggests that with more codebooks, the system better captures nuanced details that contribute to the naturalness of the audio. However, while this strategy enhances perceptual naturalness, it may not uni-
MUFFIN
1.663
0.807
2,360
0.932
4,074
4.225
Oeste (Libri-TTS)
GT
3.453
OPLS
5,390
2.703
1.143
0.695
1.271
2,815
Facodde
1.998
1.119
1.960
0.888
2.026
4.017
DAC
1.813
0.913
2,220
0.897
2.497
4.053
HiFi-CCode
1.681
0.840
2.419
0.919
3.216
4.296
Mimi A
2.515
1.688
1.611
0.612
2.498
3.679
MUFFIN
1.615
0.758
2.658
0.934
3.444
4.454
MUFFIN v
1.681
0.817
2,232
0.914
3.516
4.276
MUFFIN
1.725
0.875
2,086
0.904
3.560
4.129
tion capability on the full 12-hour IEMOCAP dataset, which contains expressive emotional speech that was not used dur- ing training. While reconstruction fidelity declines across the NACs, as shown in the objective metrics, both the default and nigh-compression variants of MUFFIN demonstrate superior robustness, achieving higher naturalness in human- perceived audio quality based on UTMOS scores compared to the ground truth references, despite the high compres- sion. Nevertheless, the prominent drop in reconstruction fidelity for emotional content highlights a challenge in pre- serving emotional nuances, which could potentially impair emotional recognition in downstream tasks.
6

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Table 2. Objective evaluation of the reconstructed speech from the IEMOCAP datasct was conducted using various neural audio codec models, following the same setup as before.
derscoring the robustness and efficiency of MUFFINS when compared to other NACs. The results consistently demon- strate the improved quality of our neural codec for general audio reconstruction.
Model
STFT
MEL
PESQ
STOI
UTMOS
VISQOL
GT
1.859
OPUS
2.361
1,586
1.207
0.478
1.242
2.642
Encodec
2,150
1,290
1.649
0.746
1.321
3,501
4.2. Ablation Studies of Deconstructing MBS Codes
DAC
L,553
0.781
1.867
0.763
1,316
3.774
Auditory feature in codebook representations. In this section, we examine the information encoded in MUFFIN's learned codebooks, which correspond to different auditory frequency bands. MUFFIN's codebooks focus on isolating the perceptual characteristics of speech attributes, guided by psychoacoustic research, and do so without the need for label-targeted supervision. To enhance understanding, we will also provide demos (Section F) of audio reconstructed from each codebook, accessible via the link.
HiFi-Codee
1.447
0.755
1,988
0.763
1,564
3.651
Mimi A
2,112
0.755
1.433
0.494
1.427
2.801
MUFFIN
1,399
0.675
2.178
0,806
1,903
4,000
MUFFIN
1.392
0.703
1.844
0.748
2.026
3.612
MUFFIN
1.429
0.754
1.726
0.723
2.019
3,376
Table 3. Objective evaluation of the reconstructed music from the
GTZAN dataset using different NACs.
Model
STFT
MEL.
PESQ
STOI
FICORR
VISQOL
OPUS
7.786
3.462
1,081
0.424
0.727
2.414
Encodec
2.712
1.016
1.684
0.756
0.882
4.247
DAC
2.493
0.928
1,709
0.741
0.867
4,220
Codebook (Low-frequency bands, 0 18.75 Hz) contains fundamental frequencies and strong harmonic content neces- sary for conveying core speech information. However, they primarily capture broad aspects of speech and lack detail in articulating speech content. This approach contrasts with previous NAC models where full-band RVQ often consoli- dates most speech information into the first codebook. To as- sess semantic content preservation in these low frequencies, we measure the STOI and word error rate (WER) using the pre-trained Whisper-large V3 model (Radford et al., 2023), which analyzes audio reconstructed by the NAC model. For compatibility with the ASR model, trained on 16 kHz au- dio, we use the LibriSpeech test-clean dataset (Panayotov et al., 2015), consisting of 2,620 samples that are resampled during reconstruction. Each sample's speech decoded from individual codebooks is processed by the ASR model. We also evaluate other NAC models, including MUFFIN with vanilla RVQ, to explore how different codebooks affect the semantic information in speech.
HiFi Codee
2.517
0.954
1.674
0.727
0.899
4.170
MUFFIN
2.360
0,866
1.815
0.760
0.896
4.298
MUFFIN
2.492
0.928
1.474
0.674
0.879
4.273
MUFFIN
2.550
0.987
1.409
0.642
0.872
4.223
Tables 3 and 4 present zero-shot reconstruction results full data samples for music and audio, specifically from the GTZAN and BBC datasets. We observe a general decrease in fidelity for music reconstruction, as the task becomes more challenging due to the need to reconstruct multiple in- strumental audio channels. From the table, we observe that HiFi-Codec achieves higher FOCORR, indicating superior pitch accuracy and suggesting that its model structure better preserves vocal quality compared to other NACs. However, the difference in FOCORR between MUFFIN and HiFi- Codec is minimal, down to the finer decimal places, and MUFFIN consistently outperforms other NACs across the remaining metrics. Moreover, while MUFFIN V achieves higher compression rate and learns to encode more ef- ficiently with decent reconstruction fidelity for music, confirmed by its competitive ViSQOL scores, we observed that PESQ and STOI were significantly lower, as reflected in Table 2. This suggests that highly compressed models face challenges in preserving fine information and are more vul- nerable to reduced generalizability in zero-shot inference.
Table 5. The table presents the WER of ASR performance on re- constructed speech from each NAC's codebook using the Whisper- large V3 pre-trained model.
MUFFIN
RVQ
Hifi-Codee
DAC
Encodoc
GT
WER: 2.41; STOI: -
WER
STOI
WER
STOI
WER
STOI
WER
STOI
WER
STOI
0,940
2.72
0.930
3.00
0.919
0.901
3.15
0.900
Table 4. Objective evaluation of the reconstructed speech from the BBC dataset was conducted using various neural audio codec models, following the same setup as before.
Code
70.3
0.644
75.6
0.702
154
0.572
36.1
0.731
33.7
0,764
Code
114
0.379
141
0.426
139
0.454
132
0.148
159
0.199
Code
191
0.436
100
0.157
100
0.090
100
0.079
153
0.121
Model
STFT
MEL
VISQOL
Code 4
107
0.082
101
0.086
112
0.129
100
0.049
147
0.094
OPUS
6.093
2.984
1.000
Encodee
1.998
1.011
3.852
Table 5 highlights the crucial role of Codebook 1 in the MUFFIN model, as it becomes significantly challenging to recognize content using any codebook other than Codebook 1, with recognition errors exceeding 100. Unlike vanilla RVQ, which typically consolidates most speech content into the first codebook, MUFFIN strategically distributes infor- mation between Codebooks and 2. The main speech intel- ligibility is allocated to Codebook 1, while details related
DAC
1.846
0.784
3.995
HiFI-Codoc
1.733
0.795
4.009
MUFFIN
1.688
0.720
4.065
MUFFIN
1,700
0.748
4.010
MUFFIN
1.706
0.777
3.997
Table 4 showcases the strong generalizability of zero reconstruction on general audio from the BBC dataset, un-
7

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
to articulation are captured in the mid-frequency range of Codebook 2, which provides it with more contextual infor- mation than Codebook 3. Notably, MUFFIN's Codebooks and 2 together achieve an STOI of 0.729 and WER of 19.2. In contrast, NACs that employ traditional RVQ often show a progressive decline in semantic content across subsequent codebooks, accompanied by a noticeable decrease in STOI.
Table 6. The zero-shot TTS results include word error rate (WER), mean opinion score (MOS) for speech quality and naturalness, and speaker mean opinion score (S-MOS) for speaker similarity, evaluated for VALL-E with three different codecs. MOS and S- MOS are reported with 95 % confidence interval.
Systems
WER (5)
MOS
S.MOS
SECS
VALL w/ Encodec
21,05
3.91 1: 0.287
3.70 1: 0.368
0.5914
VALL-E w/ Hile-Codec
32.35
4.00 4: 0.532
4.04 +: 0.333
0.5874
Codebook 3 Hligh-freuuency bands, 37.5 75 Hz) captures essential auditory cues, such speaker identity, pitch, and timbre, which are crucial for distinguishing speakers and enriching the depth of reconstructed audio. To assess the effectiveness of each codebook in preserving speaker in- formation, we randomly select 600 speech files from the Vox dataset (Nagrani et al., 2017), each representing one of six distinct speakers. Latent features are extracted from each codebook and then average-pooled to form vec- tor representation for each speech sample. Using t-SNE (Van der Maaten & Hinton, 2008), we visualize these vec- tor representations in a two-dimensional space to identify potential clusters corresponding to different speakers, as presented in Figure 5 (Appendix E,, Figure 5).
VALLE w/ MEFFIN
12.20
4.18 d: 0.278
+: 0.288
0.6099
codes autoregressively, while the remaining layers are pre- dicted non-autoregrrssively It has gained popularity as a zero-shot TTS framework and is frequently used as a bench- mark in various studies (Du et al., 2024a; Shen et al., 2023).
We follow the experimental settings in (Zhou et al., 2024b) and test 3 different VAIL-E systems on the test-clean set from LibriTTS. For each speaker in the test set, one speech sample from the same speaker is randomly selected as the prompt, ensuring is distinct from the speech to be synthe- sized. All the synthesized speech are ranging from 3 to 10 seconds. We first calculate the word error rate (WER) of the synthesized speech, as shown in Table 6. Compared to VALL-E using Encodec or Hifi-Codec, VALL-E with MUF- FIN achieves the lowest WER results, demonstrating supe- rior robustness the synthesized speech. We then report the subjective evaluation results in Table 6, where 10 native speakers evaluated a total of 80 synthesized speech samples. Each speech sample was rated on a 5-point scale for speech quality and naturalness (MOS), as well as speaker similarity to the prompt speech (S-MOS). Consistent with the objec- tive results, the VALL-E system using MUFFIN achieves the highest scores for both speech quality and speaker sim- ilarity. These findings suggest that MUFFIN effectively disentangles between semantic and acoustic information, thereby reducing error propagation during the TTS training and resulting in improved speech quality during inference.
From Figure 5, Codebook 1 exhibits a broad, dispersed dis- tribution with some cluster overlap, suggesting it encodes foundational speech content with substantial variance across samples. Codebook 2 forms a more concentrated central cluster with significant overlap between speakers, imply- ing shared features likely related to vowel and consonant articulation. Codebook 3, in contrast, shows well-separated clusters with the most distinct boundaries among speak- ers and minimal overlap, indicating its focus on capturing speaker-specific features. This distinct clustering should lead to a low distance error in speaker group classification, suggesting that Codebook 3 is highly effective at distinguish- ing between different speakers based on their unique vocal attributes. Additionally, we strongly suggest comparing the auditory results between Codebook 3 and others using the provided demos to gain a more intuitive understanding. Lastly, Codebook 4 presents compact distribution with considerable class overlap, suggesting it primarily encodes random or residual features that contribute minimally to core speech information. Additionally, the variations and the distributive patterns in the t-SNE plots suggest that each codebook captures slightly different information.
5. Conclusion
In conclusion, we introduced MUFFIN, neural psychoau- dio codec that offers a novel perspective on quantizing units with the proposed MBS-RVQ within the latent space. By strategically aligning the codec architecture with psychoa- coustic principles, MUFFIN achieves an optimal balance between compression efficiency and perceptual fidelity, tack- ling longstanding challenges in the domain. Extensive eval- uations demonstrate MUFFIN'S SOTA performance across diverse audio types reconstruction and downstream zero- shot TTS task. Furthermore, the development of a 1920 times highly compressed MUFFIN variant underscores its ability to maintain perceptual quality even under extreme compression settings. This study lays the groundwork for future advancements in real-time low latency neural audio coding and its integration with LLMs, providing a robust and scalable solution for speech- L. model applications.
4.3. Zero-shot TTS with MUFFIN tokens
We further evaluate the performance of different codec sys- tems in a zero-shot text-to-speech (TTS) setting. For this evaluation, we use VALL-E (Wang et al., 2023) the TTS framework, implementing it based popular release?, and trained it on LibriTTS dataset. VALL-E employ a decoder- only language model, predicting the first layer of acoustic
hhttpe://githubb.com http:///ohoocoo///eeeee///////// wew.ffe//taaalaleee/eeeeee
8

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Acknowledgements
complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 2002.
This research was conducted at the Alibaba-NTU Singapore Joint Research Institute.
Belkin, M. and Niyogi, P. Laplacian eigenmaps for di-
mensionality reduction and data representation. Neural computation, 15(6):1373-136, 2003.
Impact Statement
The development of MUFFIN, high-fidelity Neural Psy- choacoustic Coding (NPC) framework, represents a novel and effective alternative to conventional speech encoders (Hsu et al., 2021; Chen et al., 2022; Ng et al., 2023). This framework offers broad applicability across domains in- cluding media streaming. telecommunications, and assis- tive technologies. By incorporating psychoacoustic prin- ciples to guide perceptual compression, MUFFIN enables the transmission of high-quality audio at significantly lower bitrates. This advancement facilitates wider accessibility in sandwidth-conssrainee environments, thereby promoting digital inclusion and improving access communication and entertainment services in regions with limited internet infrastructure. In addition to its compression efficiency, MUFFIN introduces mechanism for disentangling speaker identity from speech content. This capability offers new op- portunities for personalization and content manipulation but also raises critical ethical considerations related to privacy, data security, and the potential for misuse in synthetic or ma- nipulated speech, such as deepfake audio. These concerns highlight the urgent need for responsible deployment prac- tices and appropriate regulatory oversight. Moreover, the integration of MUFFIN with large-scale language models has the potential to significantly enhance human-computer interaction in areas such as creativity support, education, and accessibility-driven applications. This synergy posi- tions MUFFIN as foundational component in the next generation of Al-powered multimedia systems. Future search should focus on developing robust bias mitigation strategies and establishing clear ethical frameworks to guide the deployment of neural audio coding technologies in a fair and secure manner across diverse global populations.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv: 2013.
Bosi, M., Brandenburg, K., Quackenbush, S., Fielder, L., Akagiri, K., Fuchs, H., and Dietz, M. Iso/iec mpeg-2 advanced audio coding. Journal the Audio engineering society, 45 45(10):78988) 1997.
Brandenburg. K. and Bosi, M. Overview of mpeg audio: Current and future standards for low bit-rate audio coding, Journal of the Audio Engineering Society, 45(1/2)44-1,, 1997.
Busso, C., Bulut, M., Lee, C.-C., Kazzemzdeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335- 359, 2008.
Byun, J., Shin, S., Sung. J., Beack, S., and Park, Y. Opti- mization of deep neural network (dnn) speech coder using a multi time scale perceptual loss function. In Interspeech, pp. 4411 1415,, 2022.
Chen, H.-.., Meng. Y., and Lec, H.-y. Once-for-all se- quence compression for self superviue speech models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Chen, J., Dai, Z., Ye, Z., Tan, X., Liu, Q., Guo, Y., and Xue, W. Pyramidcodec: Hierarchical codec for long- form music generation in audio domain. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4253 2024.
References
Ai, Y., Jiang, X.-.., Lu, Y.-X., Du, H.-P., and Ling, Z.- H. Apcodec: A neural audio codec with parallel ampli- tude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402. 10533, 2024.
Chen, S., Wang, C., Chen, Z., WU, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large- scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1005-1511 2022.
Alsteris, L. D. and Paliwal, K. K. Short-time phase spectrum in speech processing: A review and some experimental results. Digital signal processing, 17(3):578-61) 666,, 2007.
Cover, T. M. Elements of information theory. John Wiley & Sons, 1999.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gen- crative adversarial networks. In International conference machine learning, pp. 214 213. PMLR, 2017.
Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405. 2/060, 2024.
Bartlett, P. L. and Mendelson, Rademacher and gaussian

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. arXiv preprint 2XXiv:2210.13336 2022.
Hsu, W.-N, Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 19:3451-3400 2021.
Défosse7, A., Mazaré, L.., Orsini, M., Royer, A., Pérez, P. Jégou, H., Grave, E., and Zeghidour, N. Moshi: speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.000.77 2024.
Jadoul, Y., Thompson, B., and De Boer, B. Introducing parselmouth: A python interface praat. Journal of Phonetics, 71:1-15, 2018.
Du, Z., Chen, Q. Zhang, S., Hu, K., Lu, H., Yang, Y.,
Johnston, J. D. Transform coding of audio signals using perceptual noise criteria. IEEE Journal on selected areas in communications, 6(2):314-323, 1988.
Hu, H., Zheng, S., Gu, Y., Ma, Z., Gao, Z., and Yan, Z. Cosyvoice: A scalable multilingual zero-shot text-to- speech synthesizer based supervised semantic tokens, 2024a.
Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang. D., Liu, Y., Leng, Y., Song, K., Tang, S., et al. Naturalspeech Zero-shot speech synthesis with factorized codec and dif- fusion models. arXiv preprint arXiv:2403.03100 2024.
Du, Z., Zhang, S., Hu, K., and Zheng, S. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEFE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 591-595. IEEE, 2024b.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-17033, 2020.
Dunn, H. and White, S. Statistical measurements on conver- sational speech. The Journal of the Acoustical Society of America, 11(3):278-288, 1940.
Kubichek, R. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE pa- cific rim conference communications computers and signal processing, volume 1, pp. 125-128. IEEE, 1993.
Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., and Pons, J. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404. 10301, 2024.
Kumar, K., Kumar, R., De Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., De Brebisson, A., Bengio, Y., and Courville, A. C. Melgan: Generative adversarial net- works for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
French, N. R. and Steinberg. J. C. Factors governing the intelligibility of speech sounds. The journal of the Acous- tical society of America, 19(1)900-119, 1947.
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Ku- mar, K. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Sys- tems, 36, 2024.
Hager, W. W. Lipschitz continuity for constrained processes. SIAM Journal Control and Optimization, 17(3):321- 338, 1979.
LAION-AL BBC Sound Effects Library data card. https: //github. thub. com aot-nAAA//doodattteeaaa blob /mai//ainaaee maaain/aaaaaaaaaaaaaaaaaaaaaaaaa wwwd//ac.dccceeeee md, 2022, Accessed: [October, 7, 2024].
Han, D., Wang, Z., Xia, Z., Han, Y., Pu, Y., Ge, C., Song, J., Song. S., Zheng, B., and Huang. G. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605. 2024.
Langman, R., Jukić, A., Dhawan, K., Koluguri, N. R., and Ginsburg, B. Spectral codecs: Spectrogram- audio codecs for high quality speech synthesis. arXiv preprint arXiv:2406.05200 2024.
Herre, J. and Dick, S. Psychoacoustic models for perceptual audio coding- a tutorial review. Applied Sciences, (14): 2854, 2019.
Lee, S.-g.. Ping, W., Ginsburg, B., Catanzaro, B., and Yoon, S. Bigvgan: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2022.
Hines, A., Skoglund, J., Kokaram, A. C., and Harte, N. Visqol: an objective speech quality model. EURASIP Journal on Audio. Speech, and Music Processing. 2015: 1-18, 2015.
Lim, J. H. and Ye, J. C. Geometric gan. arXiv preprint arXiv: /705 700.002994 2017.
10

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Liu, Q. Wang, W., Jackson, P. J., and Tang. Y. A perceptually- weighted deep neural network for monaural speech enhancement in various background noise condi- tions. In 2017 25th European Signal Processing Confer- ence (EUSIPCO), pp. 1270-1274. IEFE, 2017.
Qian, K., Zhang, Y., Gao, H., Ni, Lai, C.-I., Cox, D., Hasegawa-Johnson, M., and Chang, S. Contentvec: An improved self-supervised speech representation by disen- tangling speakers. In International Conference on Ma- chine Learning, pp. 18003-18017. PMLR, 2022.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11998 2022.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, Robust speech recognition via large- scale weak supervision. In International conference on machine learning, pp. 28492-28518, PMLR, 2023.
Luo, Y. and Yu, Music source separation with band-split mn. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1893-1901, 2023.
Richter, J., Wu, Y.-C., Krenn, S., Welker, S., Lay, B., Watanabe, S., Richard, A., and Gerkmann, T. Ears: An anechoie fullband speech dataset benchmarked for speech enhancement and dereverberation. arXiv preprint wwXi:22406.66185, 2024.
Luo, Y., Yu, J., Chen, H., Gu, R., and Weng, C. Gull: A generative multifunctional audio codec. arXiv preprint 0rX1v:2404.04944, 2024.
Rix, A. W., Beerends, J. G., Hollier, M. P., and Hekstra, A. P. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international con- ference on acoustics, speech, and signal processing. Pro- ceedings (Cat. No. 01CH37221), volume 2, pp. 749 -522. IEEE, 2001.
Moore, B. C. and Glasberg, B. R. Suggested formulae for calculating auditory-filter bandwidths and excitation patterns. The journal of the acoustical society of America, 74(3):750-753, 1983.
Nagrani, A., Chung, J. S., and Zisserman, A. Voxceleb: large-scale speaker identification dataset. arXiv preprint 1rXiv:1706.08612, 2017.
Sacki, T., Xin, D., Nakata, W., Koriyama, T., Takamichi, S., and Saruwatari, H. Utmos: Utokyo-sarulab sys- tem for voicemos challenge 2022. arXiv preprint
Ng, D., Zhang, R., Yip. J.Q.. Q.. Yang, Z., Ni, J., Zhang, C., Ma, Y., Ni, C., Chng. E. S., and Ma, B. De" hubert: Disentan- gling noise in a self-supervised model for robust speech recognition. In ICASSP 2023-2023 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
arXiv:2204.02152, 2022.
San Roman, R., Adi, Y., Deleforge, A., Serizel, R., Syn- naeve, G., and Défossez, A. From discrete tokens to high-fidelity audio using multi-band diffusion. Advances in Neural Information Processing Systems, 36, 2024.
Nishimura, Y., Hirose, T., Ohi, M. Nakayama, H., and Inoue, N. Hall-e: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis. arXiv preprint arXiv:2410.044 043080 2024.
Santana, A. P., Pinhelli, F., Donini, J., Catharin, L., Man- golin, R. B., Feltrim, V. D., Domingues, M. A., al. Music4all: A new music database and its applications. In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 399 404. IEEE, 2020.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain au- dio books. In 2015 IEEE international conference on acousties, speech and signal processing (ICASSP), pp- 5206-5210. IEEE, 2015.
Shen, Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. Naturalspeech Latent diffusion models are natural and zero-shot speech and singing syn- thesizers, 2023. URL https: /tt////axiii /atpaaixvvvvoro.//.aa abs/ 2304 09116.
Petermann, D., Jang, and Kim, M. Native multi-band audio coding within hyper-autoencoded reconstruction propagation networks. In ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Sturm, B. I. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv: 1:::33300 100.. 1461, 2013.
Piczak, K. J. Esc: Dataset for environmental sound classi- fication. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Taal, C. H., Hendriks, R. C., Heusdens, R., and Jensen, J. A short-time objective intelligibility measure for time- frequency weighted noisy speech. In 2010 IEEE inter- national conference on acoustics, speech and signal pro- cessing, pp. 4214 4217 IEFE, 2010.
Pulkki, V. and Karjalainen, M. Communication acoustics: an introduction to speech, audio and psychoacoustics. John Wiley & Sons, 2015.

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Van Den Oord, A., Vinyals. O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Zhang, X., Zhang, D., Li, S., Zhou, Y., and Qiu, X. Speech- tokenizer: Unified speech tokenizer for speech large lan- guage models. arXiv preprint arXiv:2308. 1tti::300..6669999 16692, 2023b.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Zhang. X., Zhang, D., Li, S., Zhou, Y., and Qiu, X. Speech- tokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations, 2024.
Variš, D. and Bojar, O. Sequence length is a domain: Length- based overfitting in transformer models. arXiv preprint arXiv:2109.07276, 2021.
Zhen, K., Lee, M. S., Sung, J., Beack, S., and Kim, M. Psychoacoustic calibration of loss functions for efficient end-to-end neural audio coding. IEEE Signal Processing Letters, 27:2159 2163, 2020.
Wang, C., Chen, S.. Wu, Y., Zhang. Z., Zhou, L., Liu, S., Chen, Z.., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02211, 2023.
Zhou, K., Zhang. Y., Zhao, S., Wang, H., Pan, Z., Ng, D., Zhang. C.,, Ni, C., Ma. Y., Nguyen, T. H., al. Emo- tional dimension control in language model-based text-to- speech: Spanning broad spectrum of human emotions. arXiv preprint arXiv:2409. 16681, 2024a.
Wu, H., Chung, H.-I.., Lin, Y.-C., Wu, Y.-K., Chen, X., Pai, Y.C.., Wang, H.-H., Chang, K.-W., Liu, A. H., and Lee, H.y Codec-superb: An in-depth analysis of sound codec models. arXiv preprint arXiv:2402. 2024.
Zhou, K., Zhao, S., Ma, Y., Zhang, C., Wang, Ng. D., Ni, C., Hieu, N. T., Yip, J. Q., and Ma, B. Phonetic enhanced language modeling for text-to-speech synthesis. In Proc. INTERSPEECH, 2024b.
Wu, C.C.,, Gebru, I. D., Marković, D., and Richard, A. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Ziyin, L.,, Hartwig, T., and Ueda, M. Neural networks fail learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 33:1583-1594, 2020.
Xiao, W., Liu, W., Wang, M.. Yang. S., Shi, Y., Kang, Y., Su, D., Shang, S., and Yu, D. Multi-mode neural speech coding based on deep generative networks. In Proc. Interspeech, pp. 819 803, 2023.
Zwicker, E. and Fastl, H. Psychoacoustics: Facts and mod- els, volume 22. Springer Science & Business Media, 2013.
Yang, D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Y. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint 2xXv:23305.02765 2023.
Zwicker, E. and Terhardt, E. Analytical expressions for critical-band rate and critical bandwidth as a function of frequency. The Journal of the Acoustical Society of America, 1555:1533- 1980.
Yip. J. Q., Zhao, S., Ng, D., Chng, E. S., and Ma, B. To- wards audio codec-hased speech separation. In Proc. Interspeech 2024, pp. 2190-2194, 2024.
Yu, W. and Wang, X. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.0799/ 2024.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2022.
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z. and Wu, Y. Libritts: A corpus de- rived from librispeech for text-to-speech, arXiv preprint arXiv: 1904. 02880, 2019.
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang. P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross modal conversational abilities. arXiv preprint arXiv:2305./ /0000 2023a.
12

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
A. Psychoacoustic evidence of perceptual speech characteristics
PER CHR
CYGLE
PATTAAUE PCR CYCLE
30
WATT
AANERAGE WEN AND HOMEN OF
MEY-
C-VIVE WOMEN CUNN AND WHITE
DATA
100
500
1000
5000
10000
200
FREDUENCY IN CYCLES PCR SECOND
PREQUENEP CYCLES SCCONC
/REDUSIC OIGLE RECRSS
Fig. 1
Fig. 2
Fig. 3
100
1000
ARTICIAL IRTICULATTON ATTOW 1
SYILLAM AALLABLE ART ARTICUL AT -%6
DVULLBBLE ARTICULTTION JTIOO-
80
1500
RESPONSC -00
0-200Ag
CYCLES
+.10
-0
110
OUT
CYCLES
THOTTOLEPHOIIC RESPONDE OF TEST CIRCUIT WITHOUT FILTERS-00 08
ORTHOTELEPHONIC RESPONSE 1000 CYCLES OF TEST CIRCUTT WITHOUT FILTERS-DO
FREQUENCY
CYCLES PER SCCOND
Fig. 4
Fig.
Fig. 6
Figure 3. The figures have been sourced from French Steinberg (1947), which discusses how speech sounds recognized the ear. The data were collected from microphones based on human speech and then analyzed with computational tools to derive the intensity and sound pressure levels. (1) Comparison of Speech Spectra. (2) Idealized Long Average Speech Spectrum pressure of speech at 30cm from lips. (4) Articulation test with low pass filters. (5) Articulation test with high pass filters. (6) Syllable articulation versus cut- frequeney.
from lips. (3) R.m.s.
From the presented illustration, becomes clear that the core components of speech intelligibility predominantly emphasized within the lower frequency bands where the majority of vocal energy is concentrated. Fig. I demonstrates the variation in intensity levels across the frequency spectrum, measured in decibels relative to a standard sound pressure level. This figure contrasts intensity levels at a near-field distance, merely two inches from the speaker's lips, with adjustments made for gender differences in voice power and frequency content. The data reveal a pronounced convex curve with higher intensities noted in the lower frequencies, which gradually decrease as the frequency increases. This pattern supports our initial assertions and is corroborated by Fig. 2 and 3. These figures extend the observations to longer ranges (1 meter) and include measurements of RMS pressure, following the experimental setup described by Dunn & White (1940) for six male subjects.
Then, Fig. 4 and 5 present the results from articulation tests employing low-pass and high-pass filters across varying cutoff frequencies, demonstrating the impact of frequency range restrictions on speech intelligibility. Notably, a reduction in the cutoff frequency through low-pass filters correlates with a decrease in intelligibility, highlighting the critical role of higher frequencies frequencies are pivotal for capturing the intricate details and nuances that enhance speech clarity and comprehensibility:
recognition of consonants and the differentiation similar monilar-sounding syllables. contrast, high
Interestingly, although high-pass filtering up specific cutoff frequency yield improvements in speech intelligibiliity performance degrades when lower frequencies are excessively attenuated. This trend underscores the critical role of low-frequency prrionnnltt--immiillyy conveyed through vowel sounds- in preserving the spectral power and tonal richness of speech. These vowel sounds are fundamental to intelligibility, as they provide essential acoustic energy and rhythmic structure.
Fig. integrates the findings from Fig. 4 and 5, combining the effects of high- high-pass and low-pass filtering to offer more
13

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
holistic view of how frequency components influence speech intelligibility across various system settings. The intersection of the filter curves in Fig. 6 identifies critical bands, pinpointing crucial frequency range central to maintaining speech intelligibility. This analysis elucidates that speech articulation predominantly occurs within the mid-frequency bandwidth, providing key insights into the trequency-dependen nature of speech processing.
B. Validity of MBS-RVQ at latent representation space.
We note the theoretical distinction in performing band splitting at the input space versus applying to the latent representa- tions of autoencoder. This raises key considerations regarding the preservation of psychoacoustic properties within the latent representation space. However, we argue that latent representations retain the psychoacoustic properties of speech is supported by the Lipschitz continuity of the encoder.
Formally, if f IR" + I is Lipschitz continuous with constant L, then for any two speech signals X1,X2 (Hager, 1979):
- - 2 L x -2
Psychoacoustic cues such formant positions, harmonic relationships, and energy distributions in critical frequency bands are primarily reflected in subtle variations of the speech signal's 's waveform. Lipschitz continuity ensures that these modest yet perceptually crucial differences are neither excessively magnified nor erased when the signal is transformed into the latent domain. In other words, two psychoacoustically similar speech signals cannot become drastically separated in latent space (Arjovsky et al., 2017; Bartlett & Mendelson, 2002). Furthermore, empirical evidence in representation learning supports this notion, demonstrating that neural networks constrained by Lipschitz continuity typically learn more structured representations, within which subtle perceptual attributes remain discemible (Belkin & Niyogi, 2003; Gulrajani et al., 2017).
Consequently, if autoencoder's encoder maintains Lipschitz continuity, the latent embeddings it generates for speech signals can be expected to closely reflect the psychoacoustic characteristics present in the original waveform. Minor spectral changes perceived by listeners, such as slight shifts in vowels or sibilants, correspond to small changes in latent space, thus helping to preserve the overall psychoacoustic signature of the speech. This argument provides the basis for the notion that, although a strict one-to-one psychoacoustic fidelity is not mathematically guaranteed, in practice, Lipschitz. continuity significantly mitigates the risk of losing important auditory details in the encoder's output.
In this work, we introduce the MUFFIN encoder, a novel architecture that is provably Lipschitz continuous. The core design leverages primarily linear components-namell convolutional layers and fully connected (linear) alyrssaalongside modified Snake activation function. Convolutional and linear layers are intrinsically linear transformations, ensuring that the overall network adheres to Lipschitz continuity.
For modified snake activation function, f(x) = x + 1 (sin((az) + y the derivative is presented as:
a
3 d [x sin " (ax) + 7 =1 233 sin(ax) cos(ax)) 1 + -isin (2az)..
((())
dx
a
Since sin(2az) is bounded between -1 and 1, we have
[f(x)) = 1B sin < [8].
A function whose derivative is bounded by T. is L-Lipschitz. Therefore,
[f())) < |8]
f is Lii-sihhh
Hence, modified snake activation function is Lipschitz continuous with Lipschitz constant [8]. This property ensures stability in our model, as the layer computations with the activation functions preserve the distances between input features.
C. On the effect of MBS-RVQ compared to vanilla RVQ.
In this section, we conduct an ablation study by comparing the performance with disabling multi-band spectral residual vector quantization, reverting to vanilla residual vector quantization, to assess the contribution of MBS-RVQ in enhancing
14

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
generative quality under the constraint imposed perceptual entropy. In addition to the results presented in Table which analyze the individual contributions of each codebook and the full MBS-RVQ quantizer configuration with respect word eтror rate and short-time objective intelligibility, we further report detailed reconstruction performance of the neural psychoacoustic codec. The following tables present reconstruction quality evaluated using the same metrics employed in the main results. Complementary to the table-based evaluation, we also provide audio samples on our demo page to audibly highlight the differences between MBS-RVQ and vanilla RVQ.
Table The table presents the objective evaluation of the reconstructed speech from listed evaluation set using full codebook quantizers of MBS-RVQ versus vanilla RVQ.
Test-Clean (LibriTTS)
Model
STFT
MEI.
PESQ
STOI
UTMOS
ViSQOI.
MUFFIN
1,555
0.692
2.996
0.954
4.017
4,516
Vanilla-RVQ
1,627
0.768
2.856
0.940
3.875
4,328
MUFFIN (12.5 Hz)
1.663
0.807
2.360
0,932
4.074
4.225
Vanilla- Vanilla-RVQ (12.5 Hz)
1.755
0.879
2.260
0.924
3.785
4.017
Test-Other (LibriTTS)
MUFFIN
1.61.5
0.758
2.658
0.934
3.444
4,454
RVQ
1,683
0.810
2.544
0.917
3.318
4.268
MUFFIN (12.5 Hz)
1.725
0.875
2.086
0.904
3.560
4.129
RVQ (12.5 Hz)
1.863
0.963
1.940
0.815
3.399
3.993
IEMOCAP
MUFFIN
1.399
0.675
2.178
0.806
1.903
4,000
RVQ
1,510
0.793
2.039
0.715
1.805
3.883
MUFFIN (12.5 Hz)
1,429
0.754
1.726
0.723
2.026
3.612
RVQ (12.5 Hz)
1.584
0.835
1.644
0.645
1.917
3.455
D. Periodic Activation Function Modified Snake Activation Function
In the course of modeling, Figure depicts four distinct data scenarios, each presenting challenging frequency details where periodic patterns appear odd and highly abrupt, complicating the modelling. It is observed that the standard activation function inadequately models regions exhibiting high-frequency patterns, likely due to trade-off's involving amplitude preservation. Although the introduction of the amplitude, 3 parameter, partially mitigates this issue, it simultaneously introduces regions of elevated variance, which compromise overall stability. In contrast, observe that incorporating bias term 7 provides more robust solution by effectively stabilizing the model and reducing the variance associated with overestimating or underestimating outcomes.
In addition to the figure above, we present an ablation study examining the impact each additive term on reconstruction quality in Table 8. These results are evaluated using the same metrics as those employed in the main table and the table elucidates the distinct contributions of each additive term the reconstruction quality of the codec (i.e., the component with amplitude and bias).
Table The table presents the objective evaluation of the reconstructed speech from Test-Clean (LibriTTS) with additive term of the modifications to the vanilla snake activation.
Model (MUFFIN)
STFT
MEL
PESQ
STOI
UTMOS
ViSQOL
Added amplitude bias (Ours)
1.555
0.692
2.996
0.954
4.017
4.516
Added amplitude
1.603
0.744
2.928
0.945
3.943
4.448
Vanilla
1.635
0.760
2,876
0.940
3.905
4.409
15

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Vanilla
Added amplitude
Added amplitude & blas
(a 1.5) ( 3.0)
15) ( 1.5)
1 2 (ax) a
f a
(axx
x f
(axx) +
a 0.5
3.0
8 = 1.0
B 3.0
y= (-1,0)
x value
1.5
value
8 = 1.5
x value
(-2,0)
Data Modelling (Application)
Modified Snake(x)
Activation
sin (ax)
Data 1
Data 2
B
x sin (ax)
a
x + - sin (ax)
+
Ground Truth (data)
Data 3
Data 4
Figure 4. Illustration of our proposed modifications to the vanilla snake activation and its behavior in actual modeling for different sequential data.
E. Illustrations of the auditory feature across various codebook representations.
In this section, conduct comprehensive examination of the perceptual characteristics inherent each distinct quantized frequency band, as revealed by the learned codebooks without target supervision. We note that the cutoff for the frequency band split follows the relative logarithmic scale of the latent sampling rate, according to the psychoacoustic studies. Our analysis utilizes multiple visualization tools to elucidate these characteristics, which include t-SNE plots, randomly sampled spectrograms, and elbow plot.
These visualizations derived from the test-clean of LibriSpeech and subset of popular speaker recognition VoxCeleb dataset, allowing explore the specific semantic and speaker traits captured within the learned representations, The combined use of these diverse methodologies not only underscores the discriminative power of the representations but also enhances our understanding of their underlying structure and variability. However, it is important to note that the latent average-pooles representations of each sampled utterance were not specifically trained for the speaker recognition (classification) task. Consequently, the optimization objectives did not aim to achieve highly deterministic speaker vectors rather to encode sequential acoustic and semantic content. This may inherently limits zero-shot performance the system on deriving the vector representations for each utterance.
In Figure 5, we justify that among all frequency bands, those in the high- gigh-freuuency range of 37.5 75 Hz distinctly demarcate speaker boundaries with minimal overlap, indicating that codebook 3 naturally disentangled speech information to quantize speaker information without target supervision. This observation is further substantiated by the distances between clusters the t-SNE plot, where these high-frequency representations are the furthest apart based on the coordinate axis compared to those from the low frequency range of 18.75 Hz (codebook 1), mid frequency range of 18.75 37.5 Hz (codebook 2), and the residuals (codebook 4).
16

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
Codebook 1
Codebook 2
15
CHTRONTTE
50M4 23
555 Oumponent
Codebook 3
Codebook 4
Figure t-SNE plot showcasing each codebook, with speech randomly sampled from VoxCeleb, effectively represents six distinct speakers of the color code.
Furthermore, we stress that the reconstructed audio from the demos presented in Section (F) offers compelling evidence that Codebook 3 effectively quantizes speaker attributes from the phonetic content. This is demonstrated by the permutation of the reconstructed audio from different codebooks excluding Codebook 3, bound to a uniformly flat pitch. Such audio remains invariant to both gender and distinct speaker characteristics, underscoring the unique role of Codebook in capturing these nuances. This resulting outcome can also be used potentially obtaín data samples for normalizing speaker speech, serving the pre-training of applications such ContentVec (Qian et al., 2022).
Likewise, in Figure 6, present the spectrogram of randomly sampled speech utterance from the LibriSpeech dataset, alongside its decomposition into different codebooks. This reconstruction incrementally utilizes quantized codebooks. The spectrogram clearly illustrates the previously mentioned flat pitch contours, showing no significant variation from the original waveform when Codebook 3 is omitted from the reconstruction. This effect is evident when only Codebooks and are utilized.
From the spectrogram, we also observe that the frequency of formants is emphasized when combining Codebooks and which reveals clearer phonetic articulation patterns distinguishing vowels and consonants. This observation allows to appreciate that Codebook 2 is responsible for quantizing the articulation and respiratory paralanguage (formant) of the speech.
Next, we plot an elbow curve of the Word Error Rate (WER) as codebooks are added incrementally in Figure 7, illustrating the contribution of each codebook to preserving speech content. A noticeable decrease in the enror rate indicates enhanced clarity and intelligibility of speech, attributed improved articulation. We posit that Codebook does not focus on
17

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
"BUT NOW HERE IS A SUBJECT OF WHICH
YOU WILL WONDER AT FIRST WHY TURNER DREW IT AT ALL"
Original waveform
Codebook
Codebook +
Codebook +
Figure 6. An illustration depicts randomly sampled speech utterance alongside its reconstruction using incremental codebooks.
MUFFIN
70
Word error rate (WER %)
60
40
30
20
10
Godebook (inclusion of previous codebooks)
Figure 7. The elbow plot of the word error rate from whisper-large model, utilizing the same setup of incremental codebooks.
contextual speech content; using Codebook alone results in high recognition errors, suggesting its limited contribution to
18

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
core speech intelligibility, while the quantized information in codebook 4 is simply error residual. This analysis, along with auditory demos in Section F, further supports the assertion that Codebooks and 2 are primarily responsible for enhancing speech intelligibility and articulation details.
Consequently, our neural psychoacoustic coding with MUFFIN offers a novel perspective that facilitates the natural disentanglement of speech attributes, guided by label-free psychoacoustic studies. This uniquely positions our approach as an innovative alternative to FACodec (Ju et al., 2024), achieving similar goals of obtaining factorized information through a considerably simpler optimization process that could spur further investigation to the advancement of low-resource factorized speech representation learning.
While psychoacoustic studies have primarily focused on speech, applying similar analysis to non-stationary or transient sounds, such as those in music, is both important and intriguing. To explore this, we extended our decomposition approach to variety of musical genres, including singing, classical, jazz, and symphonic music. Consistent with the psychoacoustic framework used in speech analysis, we observed that:
Codebook I: primarily captures vocal content and coarse rhythmic beats.
Codebook 2: emphasizes vocal clarity and mid-frequency information.
Codebook 3: encodes pitch details reflective of the singer's unique characteristics.
The above characteristics are demonstrated with samples of music audio presented in our demo page. Interestingly, instru- mental content does not clearly separate across Codebooks 2 and 3, suggesting that our psychoacoustii-guidet representation is particularly effective in disentangling vocal attributes (speech and singing), but less so for purely instrumental channels. This finding reinforces the theoretical value of psychoacoustic principles for modeling vocal properties, an area that remains underexplored in neural codecs. While applying this framework to instrumental music remains challenging, we believe this opens new research directions. Further investigations, beyond the scope of the current study, may be investigated in our future work.
F. Diseussion on zero-shot text-to-speech synthesis.
F.. Training details
We evaluate the performance of MUFFIN within the VALL-E framework (Wang et al., 2023). In contrast to the original study, we train the model with fewer than 600 hours of data. Additionally, during inference, we select a random speech segment from the same speaker to use as the prompt, rather than using the first 1-3 seconds of the speech to be synthesized, as the original study did. This latter approach typically provides more consistent speaker information and constitutes an casier task. These modifications likely account for our lower performance compared the results reported in the original paper.
F.2 Discussion
During our experiments, observed that the open-sourced VALL-E configuration did not integrate effectively with MUFFIN operating at 12.5 Hz sampling rate. This discrepancy highlighted the sensitivity of prompt sequence length in learning sequential decoding information. Longer prompts tended to simplify the task, leading the model to undergeneralize, while shorter prompts provided insufficient information, causing the model to collapse prematurely. Balancing the length of prompts is crucial, particularly as each frame now encapsulates more complex information due to high compression. Consequently, noise in the output sequence reduced speech clarity. Despite these issues, the naturalness of the synthesized speech was reasonably good, demonstrating the benefits of using tokenized audio units. These units separate speech intelligibility from speaker information into distinct, independent codebooks. which are unaffected by adverse conditional computations. This observation emphasizes the importance of the speech codec's 's sampling rate in text-to-speech (TTS) systems, directly influencing the quality and intelligibility of synthesized speech. Future work will further investigate this trade-off through systematic experimentation to find optimal configurations that balance resource efficiency with high-quality speech synthesis.
Furthermore, it would also be highly beneficial to consider weakly supervised training for each psychoacoustic codebook by providing targeted labels in small volumes. Now that we have clearer understanding of the purpose of each codebook,
19

---

MUFFIN: Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding
becomes more intuitive apply appropriate supervision with minimal effort, optimizing the quality of the embedding networks while reducing the cost of collecting extensive labeled data for the disentanglement of speech information factorizing attribute codebook. We anticipate that this will further enhance performance TTS (Zhou et al., 2024a) or codec-based speech separation (Yip et al., 2024) downstream tasks.
G. Details of the hyperparameters and specifications of NACs at 24 kHz Sampling Audio
Note that the MACs (associated with real-time latency) computed based on a I-second audio waveform sampled at 24 kHz, using the tool available https://itthb..cm https//gthubb.com/vvrrssvw//i wttp:rawwwiioppscuuuueetttotoooe
The table below demonstrates that MUFFIN achieves significantly lower MACs, particularly compression increases with higher downsampling rates, compared to existing codecs. This indicates that MUFFIN offers lower latency rate than other codecs, including the baseline HiFi-Codec, thereby supporting improved real-time applications.
Num. of Params (M)
MACs (G)
Encoding
Downsampling
Frame Rate
Bandwidth
Model
Token/s
Encoder
Decoder
Total
Encoder
Decoder
Rate
Rate
(Hz)
(kB/s)
MUFFIN
34.2
11.9
46.1
14.7
16.9
(2, 4, 8)
320
75
2.7
300
MUFFIN
34.3
11.9
46.2
5.85
8.9
6,
960
1.35
150
MUFFIN
36.5
14.1
50.6
6.88
11.1
(3, 8,
1920
12.5
0.9
100
Encndee
7.43
7.43
14.9
1.51
4.10
(2, 4)
320
3.0
300
DAC
21.5
52.3
73.8
18.4
64.9
(2, 8,
320
3.0
300
Hifi-Codec
47.2
14.3
61.5
20.6
23.8
(2, 5,
320
75
3.0
300
20


Reviews:
[{"review_id": "rzXaLYILxy", "reviewer": "Reviewer_P3w1", "summary": "The paper introduces MUFFIN, a neural psychoacoustic codec leveraging Multi-Band Spectral Residual Vector Quantization (MBS-RVQ) and a modified snake activation function. By decomposing latent representations into psychoacoustically motivated frequency bands, MUFFIN optimizes bitrate allocation and achieves state-of-the-art audio reconstruction quality across speech, music, and environmental sounds. Extensive experiments demonstrate superior performance over existing codecs (e.g., HiFi-Codec, Encodec) in both standard and high-compression settings, with applications in zero-shot text-to-speech synthesis.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments and thoughtful concerns, which help to improve the impact of this work and spark further discussion.\n\n**Using larger audio set:**\nWe agree with the valid concern regarding the size of the environmental audio dataset, and we welcome further discussion on this topic. Our findings indicate that integrating both speech and music data help to overcome low-resource while achieving SOTA environmental audio performance, even when using a smaller set, comparing to various off-the-shelf models (DAC, EnCodec) trained on much larger collections (see Table 4). This can be attributed to our training that follows existing work, which uses short 1s segments that capture brief vocal or instrumental passages. These segments tend to be somehow similar in terms of their audio characteristics to some environmental audio, thereby reducing the reliance on distinctive larger audio datasets. Moreover, our interest is on the vocal, where psychoacoustic features, such as vocal timbre and articulation, are the highlight of the neural psychoacoustic codec. Thus, we will not consider using larger audio dataset in this work as compared to speech and music (covering singing vocal).\n\n**Human evaluation and latency:**\nWe agree that the absence of human evaluations is a common concern. To address this, we will add a dedicated subsection in Appendix F to explain how the objective metrics, widely adopted in the literature, correlate with human perceptual quality, thereby demonstrating the self-sufficiency of our report without human for codec. Further detailed clarifications will also be provided in response to Reviewer grYU (using SECS for evaluation). Similarly, our latency benchmarks, presented in Appendix F, are based on MACs and model parameters, which provide a good objective measure of inference time while normalizing for factors such as GPU specifications.\n\n**Transient audio:**\nWe appreciate the reviewer’s thoughtful observation. While psychoacoustic studies have primarily focused on speech, we agree that applying similar analysis to non-stationary or transient sounds, such as those in music, is both important and intriguing.\nTo explore this, we extended our decomposition approach to a variety of musical genres, including singing, classical, jazz, and symphonic music. Consistent with the psychoacoustic framework used in speech analysis, we observed that:\n\n•\tCodebook 1: primarily captures vocal content and coarse rhythmic beats.\n\n•\tCodebook 2: emphasizes vocal clarity and mid-frequency information.\n\n•\tCodebook 3: encodes pitch details reflective of the singer’s unique characteristics.\n\nWe have updated our demos to include samples that support these observations. Interestingly, instrumental content does not clearly separate across Codebooks 2 and 3, suggesting that our psychoacoustic-guided representation is particularly effective in disentangling vocal attributes (speech and singing), but less so for purely instrumental channels. This finding reinforces the theoretical value of psychoacoustic principles for modeling vocal properties, an area that remains underexplored in neural codecs. While applying this framework to instrumental music remains challenging, we believe this opens new research directions. Further investigations, beyond the scope of the current study, will be discussed in our future work section to encourage continued exploration of this promising line of research in appendix.\n\n**Practical limitation of 12.5 Hz:**\nAchieving high compression rates in audio codecs often challenges the preservation of the full spectrum of human hearing, potentially leading to muffled sounds or perceptible artifacts (especially so for streaming with reference to MiMi's performance). To address this, integrating psychoacoustic models can enhance reconstruction quality by optimizing compression across multiple frequency bands, focusing on perceptually significant components. However, implementing such models typically necessitates more complex and deeper neural networks to effectively quantize and encode the nuanced psychoacoustic information without significant loss. This increased complexity can lead to larger model sizes, which may offset the benefits of efficient compression by demanding more computational resources and storage capacity. Therefore, a careful balance must be struck between leveraging psychoacoustic properties for improved audio quality and managing the trade-offs related to model complexity and compression efficiency. This could spark more research work to investigate in this area.", "claims_and_evidence": "Yes, both the experimental data provided by the author and the audio provided on the homepage demonstrate the effectiveness of their method.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I haven't checked all the proofs because I'm not very familiar with this field and haven't found any errors yet.", "experimental_designs_or_analyses": "I checked the main experiments in the paper and refer to the section on Other Strengths and Weaknesses for detailed opinions.", "relation_to_broader_scientific_literature": "The key contributions of MUFFIN are deeply rooted in and extend the broader scientific literature on neural audio coding, psychoacoustics, and multi-band signal processing. By introducing MBS-RVQ, leveraging psychoacoustic principles, and proposing novel architectural improvements (e.g., modified snake activation), MUFFIN addresses longstanding challenges in the field and sets a new standard for high-fidelity, efficient audio compression. Its applications in zero-shot TTS and potential integration with LLMs further underscore its relevance to cutting-edge research in speech and audio processing.", "essential_references_not_discussed": "I am not deeply versed in this field, but I believe the author has provided a fairly comprehensive citation of relevant work.", "other_strengths_and_weaknesses": "Strengths: MBS-RVQ effectively disentangles speech attributes (content, speaker identity) into distinct codebooks, aligning with psychoacoustic principles. This is a significant advancement in neural audio coding. The Lipschitz continuity analysis of the encoder and ablation studies (e.g., t-SNE visualizations, codebook-specific reconstructions) validate the design choices. MUFFIN outperforms baselines across metrics (PESQ, STOI, UTMOS) and datasets (LibriTTS, IEMOCAP, GTZAN), particularly at high compression rates (12.5 Hz). The codec’s efficiency (lower MACs than HiFi-Codec) and compatibility with LLMs (via tokenized representations) highlight its potential for real-time and generative applications.\n\nWeaknesses: The ESC-50 dataset (3 hours) is small compared to speech/music datasets, raising concerns about generalizability to environmental audio. Automated MOS (UTMOS/ViSQOL) is used instead of human evaluations, which are critical for perceptual quality claims. While MACs are reduced, latency and real-time performance are not quantitatively compared to streaming-focused codecs like AudioDec. Although misuse risks (e.g., deepfakes) are acknowledged, concrete mitigation strategies are absent.", "other_comments_or_suggestions": "- Include human subjective evaluations (MOS) to strengthen perceptual quality claims.\n- Expand environmental sound experiments with larger datasets (e.g., AudioSet).\n- Discuss latency benchmarks relative to real-time codecs (e.g., OPUS, AudioDec).\n- Clarify ethical safeguards (e.g., watermarking synthesized audio) in the impact statement.", "questions_for_authors": "- How does MUFFIN handle non-stationary or transient sounds (e.g., percussive elements in music), given the focus on speech-centric psychoacoustics?\n- Could the environmental sound performance be improved with a larger dataset, or is the current approach inherently biased toward speech/music?\n- What are the practical limitations of the 12.5 Hz variant in real-time streaming, given the increased downsampling rate?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6KSQ18pNlK", "reviewer": "Reviewer_grYU", "summary": "MUFFIN is an improved RVQ-based neural audio codec (NAC) using a multi-band spectral split for each RVQ sub-layer, to better disentangle different frequency bands into separate RVQ sub-layer codebooks (\"psychoacoustically guided\"). This enables improved bitrate allocation based on psychoacoustic studies, which bridges traditional codec design (MP3, Opus) and NAC towards perceptual-oriented architectural design.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your dedication to carefully scrutinize our work and it means a lot to us.\n\n**Disabling MBS:**\nWe agree that conducting an ablation study by disabling “MBS” is important to better demonstrate its contribution to the reconstruction performance. Part of this analysis has already presented in Table 5, where we compare MUFFIN with plain RVQ to evaluate WER, STOI, and the behavior of each individual codebook. To further strengthen our empirical evidence, we will include detailed reconstruction performance results in a new appendix, as shown below focusing on speech reconstruction.\n\n\n**LibriTTS (test-clean)**\n| Model      | STFT | MEL | PESQ | STOI | UTMOST | ViSQOL|\n|-|-|-|-|-|-|-|\n| MUFFIN     | 1.555 | 0.692 | 2.996 | 0.954 | 4.017 | 4.516 |\n| RVQ   | 1.627 | 0.768 | 2.856 | 0.940 | 3.875 | 4.328 |\n| MUFFIN (12.5 Hz)     | 1.663 | 0.807 | 2.360 | 0.932 | 4.074 | 4.225 |\n| RVQ (12.5 Hz)   | 1.755 | 0.879 | 2.260 | 0.924 | 3.785 | 4.017 |\n\n\n**LibriTTS (test-other)**\n| Model      | STFT | MEL | PESQ | STOI | UTMOST | ViSQOL|\n|-|-|-|-|-|-|-|\n| MUFFIN | 1.615 | 0.758 | 2.658 | 0.934 | 3.444 | 4.454 |\n| RVQ | 1.683 | 0.810 | 2.544 | 0.917 | 3.318 | 4.268 |\n| MUFFIN (12.5 Hz) | 1.725 | 0.875 | 2.086 | 0.904 | 3.560 | 4.129 |\n| RVQ (12.5 Hz)       | 1.863 | 0.963 | 1.940 | 0.815 | 3.399 | 3.993 |\n\n\n**IEMOCAP**\n| Model      | STFT | MEL | PESQ | STOI | UTMOST | ViSQOL|\n|-|-|-|-|-|-|-|\n| MUFFIN | 1.399 | 0.675 | 2.178 | 0.806 | 1.903 | 4.000 |\n| RVQ | 1.510 | 0.793 | 2.039 | 0.715 | 1.805 | 3.883 |\n| MUFFIN (12.5 Hz) | 1.429 | 0.754 | 1.726 | 0.723 | 2.026 | 3.612 |\n| RVQ (12.5 Hz) | 1.584 | 0.835 | 1.644 | 0.645 | 1.917 | 3.455 |\n\nFrom the above tables, there is consistent improvement over using MBS demonstrating the effectiveness of MUFFIN (supported by theorem).\n___\n**Using SECS for evaluation:**\nWe acknowledge that our current evaluation metrics for the codec do not include human evaluations, and we agree that SECS may be a viable addition. The metrics we adopted follow previous work (e.g., Codec-SUPERB), which argued that these objective measures provide sufficient coverage.\n\nNevertheless, we appreciate your suggestion and have attempted it. However, given the STOI scores of the reconstruction results and the nature of our task (i.e., not generating entirely new speech but reconstructing existing ones), it can be challenging for human evaluators to reliably distinguish subtle quality differences, as shown in the demos (especially without cherry-picking samples). Similar issue has been discussed in [1].\n\nTherefore, we find it difficult to implement and believe that relying on objective metrics is more appropriate for evaluating the codec’s performance with more precise distance metrics. However, we are also careful with our evaluations and have indeed used human evaluation for our TTS outputs, as shown in Table 6, where naturalness (MOS) and speaker similarity can be meaningfully assessed. We trust that this is a common valid concern and will discuss in a new appendix section to dismiss some misunderstanding/issue with evaluating codec performance with human while evaluating/citing SECS.\n\n[1] Varadhan, Praveen Srinivasa, et al. \"Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation.\" arXiv preprint arXiv:2411.12719 (2024).\n___\n\n**Annotation of off-the-shelf models in tables and snake activation references:**\nIt makes perfect sense and we will update the manuscript with the provided references to enhance the credibility of our results. We deeply appreciate your effort in pointing out our weaknesses. \n___", "claims_and_evidence": "The claims and evidence are mostly presented adequately via quantitative and qualitative analysis, including sub-layer reconstruction and PCA analysis of each codebook. However, I would like to see an ablation study by disabling \"MBS\" portion of MBS-RVQ, while keeping the other design unchanged. In other words, having a MUFFIN model only trained on plain RVQ (EnCodec, DAC, or HiFi-Codec) without the psychoacoustic guidance as comparison would be better, since the usage of \"MBS\" is a core claim in this work, such ablation study seems to be the most important experiment.", "methods_and_evaluation_criteria": "The work follows conventional metrics in codec reconstruction, which makes sense. The addition of WER is a nice addition, as some of recent low frame-rate codes do not perform great in these metrics even though they are good at the acoustic reconstruction metrics, including UTMOS and ViSQOL. I suggest the authors to also consider SECS as a viable metric as well.", "theoretical_claims": "A theoretical claim of the psychoacoustic evidence of perceptual speech characteristics is based on a well established literature, which is not necessarily new claim but serves as a good reference to bridge the existing theory into a neural model design.", "experimental_designs_or_analyses": "Since the authors retrained HiFi-Codec with the same configuration, but not others, I think it's good to annotate it in the evaluation result tables.", "relation_to_broader_scientific_literature": "The findings can potentially give the NAC community an attention to bring psychoacoustic study which is well-studied for past dacades, to bring domain-specific knowledge into the neural design rather than disconnecting from the an estabilished past.", "essential_references_not_discussed": "BigVGAN [ICLR'23] is the first work that introduced Snake activation into the audio decompression domain (as a mel spectrogram vocoder), and in fact, it is also the first study that also proposed a learnable scaling factor β (called SnakeBeta) from its official implementation. However, current manuscript only points this to follow-up studies (DAC and Stable Audio). Since this paper introduces a further study of the periodic activation function design, I suggest the authors to include the above-mentioned original reference.", "other_strengths_and_weaknesses": "Please see Claims And Evidence and Questions section.", "other_comments_or_suggestions": "None", "questions_for_authors": "As mentioned in the claims section, while most of the paper are well structured, I feel the no-MBS ablation important to add to make the paper's core claim stronger.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LyPHccQXP", "reviewer": "Reviewer_ycMR", "summary": "This paper proposes multi-band frequency spectral residual vector quantization (MBS-RVQ) for quantizing latent speech across different frequency bands. Additionally, the results demonstrate the performance of zero-shot text-to-speech models using the proposed Neural Audio Codec.\n\n## Update after rebuttal\n\nWhile the proposed methods could enhance RVQ, the improvement is incremental and still it requires many residuals, which is a burden for downstream tasks. Recently, many codecs are designed only with a single layer such as LLASA. I could not find any advantage in terms of efficiency. I will maintain my score as it is.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and the effort you’ve put into helping us improve our presentation.\n\n**Novelty (It appears that there are some misapprehension):**\nOur quantizer fundamentally differs from plain RVQ by employing a multi-band split directly at the latent level, guided by psychoacoustic features such as content, formant articulation, and speaker characteristics. Unlike other multi-band codecs (e.g., Spectral Codec, which splits at the input level, or HALL-E and PyramidCodec, which use multi-scale down-sampling), our approach **as accurately pointed by other reviewers that it performs a spectral split in the latent space. This design allows the codec to construct psychoacoustical disentangled features, with each codebook optimized specifically for different frequency bands (supported by theory and empirical observations), clearly distinguishing from previous work**. As demonstrated in Theorem 3.1, our method not only enhances the elegance and robustness of the codec but also pushes the boundaries of performance. We believe this novel perspective is an important contribution to the community and bridge the gap of traditional work of MP3.\n\n**Fairness (It appears that there are some confusion but we are happy to clarify):**\nBaseline Alignment-We intentionally align our work to HiFi-Codec (baseline) by matching codebook used of 4. HiFi-Codec represents SOTA speech performance, making it a rigorous and relevant benchmark. Additionally, we retrained the baseline on the same dataset to ensure robust performance and to control for any potential data bias, thereby ensuring a fair comparison.\nBenchmark other work: For our comparisons with other codecs (e.g., DAC and EnCodec), we adopt the “early codebooks” approach, consistent with recent studies such as WavTokenizer, SemantiCodec, and Speech Tokenizer. MUFFIN performs well even when evaluated across other papers of same dataset, showing consistency in the literature.\n\nComputationally- RVQ (benchmarked model) mathematically encourages each codebook to be as self-sufficient as possible. Each stage minimizes its own L1/L2 reconstruction error without “looking ahead” to future stages. This independent optimization ensures that the early codebooks in any model, whether a model is trained with 4 or 32 codebooks, are directly comparable, since each stage is forced to capture as much residual information as possible.\n\nLastly, MiMi is compared with the officially reported count of 8 codebooks to further validate our results. Overall, our comparative tables are constructed on a scientifically fair basis, allowing for meaningful insights rather than merely demonstrating superior performance or any attempt to over-claim our work.\n\n**Streaming:**\nOur streaming capability is built on a fully CNN-based model that follows exactly like Encodec’s design so it will be **inaccurate to say that ours is non-causal**. Specifically, our system processes audio in small window frames (3.5 seconds) that are non-causal within each frame, i.e., using such global window context, but causal over past windows for all streaming applications. Since CNNs are fundamentally local feature extractors, they do not inherently capture global context as transformers do. This locality is advantageous in streaming applications, as it allows for more stable and consistent performance when operating under strict causal constraints. By contrast, self-attention models (wavtokenizer), although trained in a non-causal manner, must be adapted to causal computation during streaming, which can lead to instability. Our approach leverages the stability of CNNs in local processing, ensuring robust streaming even when constrained to causal operation.\n\n**Following LLASA:**\nWe fully agree with the reviewer’s concern regarding the importance of these metrics. Due to space constraints, we did include the detailed table in the original submission Appendix F, where we demonstrate their impact comprehensively with latency metric of MACs and model size as well. However, we will update the references raised and the typo in line 034.\n\n**Ablation of Snake Activation:**\nWhile modified snake activation is not our core contribution in this work, we agree that showing the ablation performance helps to improve the quality of the presentation. We will include the results in appendix C while showing the performance as below\n\n **LibriTTS (test-clean)**\n| Model      | STFT | MEL | PESQ | STOI | UTMOST | ViSQOL|\n|-|-|-|-|-|-|-|\n| Added amplitude & bias (Ours) | 1.555 | 0.692 | 2.996 | 0.954 | 4.017 | 4.516 |\n| Added amplitude   | 1.603 | 0.744 | 2.928 | 0.945 | 3.943 | 4.448 |\n| Vanilla     | 1.635 | 0.760 | 2.876 | 0.940 | 3.905 | 4.409 |", "claims_and_evidence": "Using a multi-band audio representation is not new for neural audio codecs. Specifically, Spectral Codecs [1] proposed a multi-band spectral codec that encodes disjoint mel bands separately and quantizes them using frequency-wise vector quantization. HALL-E [2] introduced a Multi-Resolution Requantization (MReQ) method to quantize the latent representation from low to high frequencies. PyramidCodec [3] quantized the latent representation hierarchically by employing RVQ on multi-scale features. Language-Codec [4] also separates the latent representation and quantizes them individually.\n\n[1] Langman, Ryan, et al. \"Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis.\" arXiv preprint arXiv:2406.05298 (2024).\n\n[2] Nishimura, Yuto, et al. \"HALL-E: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis.\" ICLR, 2025.\n\n[3] Jianyi Chen, Zheqi Dai, Zhen Ye, Xu Tan, Qifeng Liu, Yike Guo, and Wei Xue. 2024. PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4253–4263, Miami, Florida, USA. Association for Computational Linguistics\n\n[4] Ji, Shengpeng, et al. \"Language-codec: Reducing the gaps between discrete codec representation and speech language models.\" arXiv preprint arXiv:2402.12208 (2024).", "methods_and_evaluation_criteria": "The model comparison is not entirely fair because other codecs, such as Encodec, DAC, HiFi-Codec, and Mimi, were not trained using four RVQs. However, the comparison may have been conducted using only four RVQ levels for the baselines.\n\nFurthermore, while Encodec and Mimi use causal convolutional layers for streaming generation, Muffin employs non-causal convolutional layers with a greater number of layers, which makes the comparison somewhat unfair. Please discuss more details for other models.", "theoretical_claims": "This paper utilizes residual vector quantization, a well-established method.", "experimental_designs_or_analyses": "Please include details such as token rate, codebook size, codebook number, and frame rate, following LLASA [5].\n\n[5] Ye, Zhen, et al. \"Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis.\" arXiv preprint arXiv:2502.04128 (2025).", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": "[1] Langman, Ryan, et al. \"Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis.\" arXiv preprint arXiv:2406.05298 (2024).\n\n[2] Nishimura, Yuto, et al. \"HALL-E: hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis.\" ICLR, 2025.\n\n[3] Jianyi Chen, Zheqi Dai, Zhen Ye, Xu Tan, Qifeng Liu, Yike Guo, and Wei Xue. 2024. PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4253–4263, Miami, Florida, USA. Association for Computational Linguistics\n\n[4] Ji, Shengpeng, et al. \"Language-codec: Reducing the gaps between discrete codec representation and speech language models.\" arXiv preprint arXiv:2402.12208 (2024).", "other_strengths_and_weaknesses": ".", "other_comments_or_suggestions": "The ablation study for the modified snake function is not conducted. \n\nTypo on line 034: \"12.5 kHz\" might be \"12.5 Hz.\"", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents MUFFIN, a neural psychoacoustic codec that leverages:
- Multi-Band Spectral Residual Vector Quantization (MBS-RVQ) to quantize latent speech representations across different frequency bands.
- A modified snake activation function to more precisely capture fine-grained frequency details. In this regard, the authors should cite the BigVGAN paper, as noted by Reviewer grYU.

Both the perceptual quality showcased on the demo website and the quantitative evaluations demonstrate solid improvements.

Regarding LLASA and single-layer codecs:
- It's a bit unfair to treat LLASA as a reason for rejection, given that the ICML submission deadline was before the LLASA paper's arXiv release. However, the authors should acknowledge and discuss the related work highlighted by the reviewer in their final draft. While single-layer codecs are a compelling direction and may ultimately be ideal, they still have received mixed feedback from practitioners.
Decision: Accept (poster)