UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner

Authors: dongchao  yang
haohan  guo
yuanyuan  wang
rongjie  huang
xiang  li
xu  tan
xixin  wu
helen  meng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 18:07:56

Abstract: Large Language models (LLMs) have demonstrated supreme capabilities in textual understanding and generation, but cannot be directly applied to cross-modal tasks without fine-tuning. This paper proposes a cross-modal in-context learning approach, empowering the frozen LLMs to achieve multiple audio tasks in a few-shot style without any parameter update. 
Specifically, we propose a novel LLM-driven audio codec model, LLM-Codec, which transfers the audio modality into textual space by representing audio tokens with words or sub-words from the LLM vocabulary, while maintaining high audio reconstruction quality.
The key idea is to reduce the modality heterogeneity between text and audio by compressing the audio modality into the well-trained textual space of LLMs. Thus, the audio representation can be viewed as a new \textit{foreign language}, and LLMs can learn the new \textit{foreign language} with several demonstrations. In experiments, we investigate the performance of the proposed approach across multiple audio understanding and generation tasks, \textit{e.g.} speech emotion classification, audio classification, text-to-speech generation, speech enhancement, etc. Experimental results show that LLMs equipped with the LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, can perform effectively in simple scenarios, validating our cross-modal in-context learning approach.
To facilitate research on few-shot audio task learning and multi-modal LLMs, we have open-sourced the LLM-Codec model.

Main paper: 
 introduction: 
 Large language models (LLMs) (e.g., GPT-4 [2] , LLAMA [36] ) have become increasingly versatile and effective in handling diverse and complex Natural Language Processing (NLP) tasks as they scale in model size and training data. It is worth noting that the in-context learning ability of LLMs can be used to solve unseen tasks, e.g., we can provide instructions along with a few demonstrations, enabling LLMs to learn and solve new tasks. The success of LLMs inspires the development of multi-modal LLMs, naturally leading to the idea of empowering their auditory capabilities to tackle audio-related tasks. There have been notable advancements in extending the capabilities of LLMs to tackle audio understanding tasks by combining the pre-trained audio encoder (e.g. Whisper encoder [31] ) and LLMs. For instance, models like WavLLM [15] , SALMONN [35] , and Qwen-audio [8] propose training multi-modal LLMs by integrating a pre-trained audio encoder, a trainable adaptor, and pre-trained LLMs. They try to align the audio and text modalities by updating the adaptor or fine-tuning the LLMs with LORA [14] . However, previous works have limitations: (1) they primarily focus on expanding LLMs to solve specific audio tasks without leveraging in-context learning for unseen audio tasks; (2) they do not support audio generation tasks, which limits their applicability;(3) they require large-scale audio data for aligning audio and text modalities, increasing the burden of model training and data collection. Learn a foreign language for different digit, then generate corresponding number using foreign language based on instructionFigure 1 : This figure illustrates the framework of the proposed approach (UniAudio 1.5) for performing speech emotion classification and simple text-to-speech generation tasks. For each task, we prepare the instruction, demonstrations (e.g., {x 1 , y 1 , x 2 , y 2 }), and the query x q . The LLAMA 2 model is then asked to predict the corresponding result y q . Here, y q can be either text or audio.In this study, we propose a cross-modal in-context learning approach, empowering the frozen LLMs to solve any user-defined audio tasks based on a few demonstrations without any parameter update.To achieve this, we introduce a vector quantization audio codec model, named LLM-Codec, that maps the audio modality to the token space of frozen LLMs (e.g., LLAMA 2 [36] ). Our motivation is to reduce modality heterogeneity between audio and text by compressing audio data into the token space of LLMs. Given that the compressed audio and text modalities share a vocabulary, the compressed audio sequence can be treated as a new foreign language, which LLMs can learn from a few demonstration samples. Moreover, since LLMs are pre-trained on large-scale data and have discovered numerous token sequence patterns, they are well-positioned to generalize to this new foreign language. Figure 1 illustrates the integration of the proposed LLM-Codec with LLAMA 2 models for performing various audio tasks.The proposed LLM-Codec aims to compress audio data into a lexical word sequence. A desired LLM-Codec should exhibit the following properties: (1) Completeness [12] : it should recover compressed audio with minimal loss. (2) Compactness: it should encode the audio into fewer-token sequences.(3) Semantic richness: it should encode audio into semantically rich token sequences, making them easier for pre-trained LLMs to recognize. Thus, we propose a semantic-guided multi-scale residual vector quantization (RVQ) based codec. Specifically, the codec model consists of three residual VQ layers: the first layer encodes semantic information, the second encodes coarse-grained acoustic information, and the third encodes residual acoustic information. Unlike previous works [9, 52] , which encode audio data into the same granularity in each layer, we propose a multi-scale approach that encodes audio data at different granularities across layers. We are motivated by the observation that semantic-level information can be preserved with fewer tokens, while acoustic-level information requires more tokens This multi-scale approach not only shortens the token sequence length but also offers flexibility for various tasks; for instance, audio understanding tasks may only require the semantic-level VQ layer. Additionally, we design a novel semantic loss and consistency loss to enhance the training of the LLM-Codec model.We conduct experiments to validate the effectiveness of LLM-Codec in an in-context learning setting.Using the pre-trained LLAMA 2 7B model without parameter updates, we evaluate LLM-Codec on various audio understanding and generation tasks, such as speech emotion classification, audio classification, simple text-to-speech, and speech denoising. The main contributions of this work are summarized as follows:• We propose a novel LLMs-driven audio codec model, LLM-Codec, which effectively bridges the text and audio modalities. To the best of our knowledge, this is the first work to quantize the audio data into the representation space of LLMs.• We demonstrate the feasibility and potential of using the in-context learning ability of LLMs to solve unseen audio tasks, including audio understanding and generation tasks. Extensive experiments and ablation studies further validate the effectiveness of our method. 
related works: 
 Audio Codec Models Audio codec models have been widely used in the audio generation domain [50, 48, 5, 49, 46] . Historical investigations into low-bitrate parametric audio codecs began with earlier studies [21, 4] . However, the quality of these codecs typically faced limitations. Recently, advancements in neural network-based audio codecs have led to several promising developments [52, 9, 47, 25, 20] . These systems typically involve an encoder that extracts deep features from a latent space, which are then quantized and transmitted to a decoder for reconstruction. Particularly relevant to our work are the FACodec [20] and SpeechTokenizer [54] models, which explicitly model different properties of audio in different vector quantization layers. In contrast, our proposed LLM-Codec encodes audio data into a lexical word sequence, differing from these approaches.Multimodal Large Language Models Recently, there has been tremendous progress in the area of multimodal LLMs. These models use pre-trained LLMs as the base and incorporate additional input modalities, such as vision [55, 28, 51, 56, 27, 37] and audio [7, 23, 15, 53, 35, 19, 40] . In general, multimodal LLMs consist of a pre-trained LLM, a pre-trained vision/audio encoder, and a modality adaptor. These systems often require the construction of extensive multimodal datasets to fine-tune the models. In the audio modality, most previous works focus on solving speech understanding [15] or general audio understanding [35, 19] , but these models are not applicable to audio generation tasks. SpeechGPT addresses some audio understanding and generation tasks by fine-tuning all parameters and expanding the speech token vocabulary based on LLAMA. However, the speech tokens in SpeechGPT contain only semantic-level information, limiting its application to broader audio tasks, such as text-to-audio generation. Additionally, SpeechGPT does not explore the in-context learning ability to handle unseen tasks.In-context Learning In-context learning, a form of few-shot learning, allows a large language model (LLM) to quickly adapt to a specific task during inference by reviewing just a few examples provided in the prompt [6] . This approach has proven successful in both natural language [45] and visual-language tasks [3, 51, 56, 28] . In the audio domain, several advanced methods have been proposed to leverage in-context learning for solving unseen audio tasks. SALM [7] introduces speechaugmented language models using in-context learning to address speech recognition and translation tasks, demonstrating that the SALM model can also handle keyword boosting. ICL-GSLM [13] employs warmup training and prompt tuning strategies to enhance the in-context learning abilities of pre-trained speech language models [26] for unseen tasks. However, ICL-GSLM primarily focuses on audio understanding tasks and overlooks audio generation tasks. Dynamic-superb [16] employs instruction-tuning for audio understanding tasks. Similarly, [42] and [41] investigate in-context learning within the speech understanding domain. Building on the success of in-context learning in NLP [45] and vision-language tasks [51, 56, 28] , our study focuses on harnessing the in-context capabilities of frozen LLMs to address a broad range of audio understanding and generation tasks. 
overview: 
 Previous audio codec models [9, 52, 47] use a VQ-VAE [38] framework, where the audio signal is first encoded into a discrete latent space and then decoded back into audio. Because audio codec models map the audio signal into discrete token sequences, many studies [17, 48, 5] have proposed training auto-regressive (AR) language models to generate these sequences, inspired by the success of LLMs in natural language processing. However, there is modal heterogeneity between the discrete audio tokens produced by the codec models and the text tokens used in LLMs. For example, the codebooks in audio codecs and the vocabularies of LLMs are typically unconnected, making it difficult to extend well-trained LLMs to audio modalities. Although previous works [53, 34] have demonstrated the effectiveness of expanding the vocabulary of LLMs to audio tokens and updating all of the parameters of LLMs, it will cost a lot of computing resources and forget the knowledge of the text. In this part, we present a large language models-driven audio codec model (LLM-Codec), which effectively bridges the gap between audio and text modalities. LLM-Codec is built on the VQ-VAE framework, but it differs from previous work in several key ways: (1) LLM-Codec is forced to quantize the audio signal into the token space of LLMs; (2) LLM-Codec adopts a multi-scale residual vector quantization strategy to balance the completeness and compactness of codec model; (3) LLM-Codec explicitly encodes different level information in different VQ layers. The following sections provide detailed insights into LLM-Codec, with Figure 2 offering a visual depiction.For any audio x, the encoder first encodes it into latent presentations E T,d , where T denotes the number of frames, d denotes the dimension of each vector. We set 4 down-sampling layers with S = [3, 4, 5, 8] in the encoder, which results in 480 times down-sampling for audio. Then, each frame e ∈ E is passed through the quantizer, which assigns it to the closest entry in a codebook, resulting in the quantized embedding ê. Finally, the quantized feature Ê inputs into the decoder to reconstruct x. Note that we add several Transformer layers in both Encoder and Decoder part to maintain a good reconstruction performance. Refer to Appendix B to find more model structure details.We use three residual VQ layers to maintain the balance between completeness and compactness. Furthermore, we propose to set different quantization granularity in different VQ layers: we expect the first VQ layer can encode the semantic information, and such information can be saved with fewer tokens, thus an interpolation function is used to down-sample the encoder features E EQUATIONSimilarly, we also apply a down-sampling operation to E T,d 2 , we set the down-sampling step as k 2 . The features become as E T /k2,d 2. Then we pass it into the second VQ layer and obtain ÊT/k2,d 2 . Lastly, we expect the last VQ layer can preserve all of the residual acoustic information. We first obtain the residual features based on the quantized features of the first two VQ layersEQUATION)Considering the residual acoustic information is more complex and diverse, we directly apply the VQ operation to each frame without any down-sampling. By using a large down-sampling step in the encoder of codec, and applying a multi-scale VQ strategy, we can effectively reduce the number of quantized audio token sequences. In our setting, 1-second audio with a 16k sampling rate will be quantized into 57 tokens. To ensure that the first VQ layers encode semantic information, we propose incorporating a semantic loss during the training process. Furthermore, to maintain the The codebook size is 32000. Furthermore, the LLAMA codebook embedding dimension is 4096, which is too large for codec training. Thus, we apply a linear mapping to 512. In the training process, the parameters of codebooks are fixed.Our approach is based on a GAN objective, in which we optimize both the generator(it consists of encoder, quantizer, and decoder) and the discriminators. For the generator, its training loss consists of three parts: (1) reconstruction loss term; (2) adversarial loss term (via discriminators); and (3) semantic and consistency losses. In the following, we give the details of proposed semantic loss and consistency loss. Refer to Appendix B.2 to find the details of reconstruction loss and adversarial loss.Semantic loss To enhance the semantic representation ability in the first layer, we introduce a semantic loss for the first VQ layer. We expect it can encode semantic information, for example, if the input audio includes a sound event, the first layer should encode which semantic information of the sound event. Similarly, if the input audio is speech, the first layer should encode the content of the speech. To realize this target, we use a pre-trained T5-base model [32] to extract a global representation vector g for the input audio content. We use Whisper to obtain its transcriptions if the input audio is speech. If the input audio is sound, we use its audio caption label:EQUATION) Consistency loss In our early experiments, we found the training of LLM-Codec is not stable, and the model is easy to collapse. One of the reasons is that we designed a significant down-sampling rate and the codebooks are fixed in the training, which increases the training difficulty. To solve this issue, we propose a consistency loss to maintain the training stability. Specifically, we propose using a pre-trained Whisper encoder [31] to extract frame-level features w, then using these features as prior knowledge to guide the second VQ layer.EQUATION) where inp denotes the interpolation function to align the feature dimension between the quantized features and whisper features. We chose the Whisper encoder because it is trained not only on speech data but also on non-speech data. Furthermore, we do not apply this loss on the third VQ layer, because we expect the third VQ layer to encode the residual information. 
uniaudio 1.5: 
 By combining the pre-trained LLMs and the proposed LLM-Codec models, we can solve many audio tasks in a few-shot style, as Figure 1 shows. We named the system UniAudio 1.5 for the reason that Table 2 : Audio understanding task evaluation results. Task induction denotes the explanatory text that precedes the sequence of audio and text. It is intended to describe the task to the model in natural language, for example: Please answer the question. Accuracy (%) is used as the metric. For the Random guess, we calculate the average based 5 times evaluation. K shots refers to the number of distinct samples for each category, and Repeats refer to how many times we copy the prompt samples. the system can be viewed as a universal audio task solver.EQUATIONConnection to UniAudio UniAudio 1.5 is an advanced edition of the UniAudio Series [48] . Compared to its previous version UniAudio [48] , UniAudio 1.5 has the following connections and distinctions. First, goal. While both UniAudio 1 and UniAudio 1.5 aim at building a universal audio foundation model for all audio tasks, their focuses are different. UniAudio focuses on audio generation tasks, such as text-to-speech, text-to-music, singing voice generation, and so on. UniAudio 1.5 focuses on audio understanding and generation tasks by exploring the few-shot ability based on large language models. Second, architecture. UniAudio 1.5 keeps the basic components in UniAudio, such as an audio codec used to transfer the audio modality into discrete representations, and a decoder-only transformer backbone is used. However, UniAudio 1.5 leverages 1) a pre-trained LLMs to solve the audio understanding and generation tasks by in-context learning, 2) an LLM-driven audio codec to quantize the audio data into the token space of LLMs. Building a multi-modal audio foundation model that is capable of handling any audio task is the ultimate goal of the UniAudio series. In UniAudio 1.0, we show the possibility of building a universal model for different types of audio generation tasks, but it (1) cannot effectively solve audio understanding tasks and (2) cannot solve unseen audio tasks in the training or fine-tuning stages. UniAudio 1.5 shows the possibility of using pre-trained LLMs for both audio understanding and generation tasks. We believe the proposed LLM-Codec in UniAudio 1.5 builds a foundation for more advanced editions of the UniAudio Series in the future.5 Experimental Results 
llm-codec training and reconstruction performance evaluation: 
 Training data LLM-Codec is a universal audio codec model, trained on both speech and sound datasets. For speech data, we use a portion of the MLS dataset [30] , and for sound data, we use the AudioCaps dataset [22] . In total, we utilized 2k hours of audio data to train the LLM-Codec model.The details of the LLM-Codec model configuration can be found in Appendix B. For the proposed multi-scale RVQ, we set three scales, with down-sampling rates of k 1 = 4, k 2 = 2, and k 3 = 1 for each layer. We initialize the VQ layers using the vocabulary of the LLAMA 2 7B model. Evaluation data We conduct evaluation on speech and sound datasets, for speech data, we choose 200 utterances from VCTK [39] , and for sound data, we choose 200 utterances from ESC 50 dataset.Baselines We compare our model with publicly available audio codec models, including Encodec [9] and DAC [25] . Additionally, we compare it with our reproduced audio codec models, which were trained on the same dataset as LLM-Codec.Evaluation dataset We choose commonly used test datasets for each task and construct N-way-Kshot test pairs. More details about constructing evaluation samples can be found in Appendix C.1.Baselines Given the limited number of works focusing on few-shot learning for unseen audio tasks, we choose BLSP [40] as one of the baselines for audio understanding tasks. Since BLSP is fine-tuned on a continuation writing task and does not explicitly address audio classification, these tasks are considered unseen for the BLSP model. Additionally, we compare our approach with instructiontuning-based models as described in dynamic-superb [16] . For audio generation tasks, we did not find directly related works, so we report the performance of state-of-the-art specialized models.We first present the reconstruction performance comparison. Then we apply the LLM-Codec and LLAMA 2 7B model for audio understanding and audio generation tasks, to verify the ability of the proposed method. Lastly, we give the visualization of LLM-Codec to explain why it can work. We leave more experiments on Appendix D.Reconstruction performance We compare the audio reconstruction quality with previous works Encodec [9] , DAC-Codec [25] , and our baseline model. We report Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). Table 1 shows the results. Compared to previous methods, the LLM-Codec achieves better reconstruction performance while utilizing fewer tokens. More specifically, the LLM-Codec model can compress 1-second audio data into a sequence that only includes 57 tokens, which significantly reduces the sequence length. Compared to the RVQ baseline model, the LLM-Codec significantly reduces the compressed tokens, and its reconstruction performance does not significantly decline. In Section 5.3, we will show the importance of compressing audio into fewer tokens. We also conduct experiments to validate whether we can use a few VQ layers, such as 1 VQ layer or 2 VQ layer, we can see that the reconstruction performance will significantly drop. To maintain the balance between completeness and compactness, we choose a multi-scale 3 VQ layer as the default setting. Speech Emotion Classification The speech emotion classification task [10] aims to predict the emotion label of the speech. We conduct 2way K-shot experiments on the ESD [1] dataset. Experimental results are shown in Table 2 . We have the following findings: (1) Task induction is important to maintain the stability of performance, we can see that without task induction, the classification accuracy will dramatically decline.(2) The semantic layer effectively extracts the global semantics of audio, which can be easily understood by the LLAMA model. (3) Using more demonstration samples (e.g. 3 shots), the performance will be better. ( 4) Repeating the demonstration samples can also bring improvement. (5) Compared to the BLSP, our method performs better in any setting. Furthermore, we also note that the performance of BLSP will drop when repeat operation is used. One possible reason is that BLSP only learns the translation relationship between text and speech, repeating samples cannot bring new cues for LLMs to solve the new task. Instead, our LLM-Codec learns to map the audio data into the latent space of LLMs, increasing the number of demonstration samples can help LLMs to find special patterns to solve this new task.Sound Event Classification Sound event classification aims to recognize the sound event in the audio.In general, an audio may include multiple events. To simplify the recognition difficulty, we assume each audio only includes one event. We conduct experiments on the ESC50 dataset [29] , which includes 50 different types of events. We construct 2-way-K-shot and 3-way-K-shot evaluations based on the ESC50 test set. Compared with the BLSP model, our proposed method gets better performance. Based on the experimental results from two audio understanding tasks, we can see that the semantic VQ layer is very important for understanding tasks. Dynamic-SUPERB Benchmark We also conduct experiments on Dynamic-SUPERB Benchmark tasks [16] . In [16] , authors propose an instruction-tuning strategy for multi-modal LLMs. They first construct a lot of audio tasks as training data, then validate some unseen audio tasks in a zeroshot way. To make a fair comparison, we use the same test set with them, and choose the first N samples as the demonstration to construct a N-way-1-shot evaluation. As Table 3 shows 4 selected audio understanding tasks, our proposed method obtains better or compared performance than these baselines in [16] . Especially, for the bird sound detection task, our proposed method obtained great improvement over previous methods. We also note that our method performs worse on language identification, the possible reason is that our codec model is only trained on English speech data. In the following, we will show that LLM-Codec also can be used to conduct audio generation tasks.We conduct text-to-speech generation on the Free Spoken Digit Dataset (FSDD) dataset [11] , which includes 3 speakers and 1,500 recordings. Unlike the traditional TTS model, which generates any speech content, this task generates digit speech. Our current model to generate complex speech content is still challenging. We use accuracy (ACC) to assess the content of the generated sample whether following the instructions. DNSMOS is used to assess the speech quality of generated samples. We construct 20 different query questions, including addition, subtraction, multiplication, division, and reasoning (finding more details from Appendix C.2). From Table 4 , we can see that our proposed model can accurately understand the query in most cases (the accuracy is 70 %) and generate good-quality speech samples. Figure 3 gives a visualization of generating speech based on the query. The frozen LLAMA model learns about 4 digits (0-3), each audio digit includes 5 samples. We add the context for each audio: "an audio of k" before inputting the audio's discrete representations into LLAMA, as Figure 1 (b) shows. After that, we let the LLAMA 2 model generate corresponding speech digits based on the instruction. We also note that the generated audio appears different from all context audio samples, demonstrating the cross-modal reasoning capability of LLMs when using the LLM-Codec as the connector for text and audio. 5 shows, we can see that the proposed method can also learn to denoise without any training. Furthermore, we also note that the performance has a large room to improve compared to special models. Token Visualization We visualize the tokens produced by the first VQ layer of LLM-Codec for different types of sound in Figure 4 . We have the following findings: (1) Although the two audios include the same sound event, the quantized sequence is not exactly the same. (2) The quantized sequence of two same types of audio has a similar pattern, e.g. their token sequences have similar repeating patterns or the same word. Such patterns may help the LLMs recognize the type of audio.Query: an audio of (0*2) Query: an audio of (1+1) Query: an audio of (1-1)Figure 3 : Examples of simple text-to-speech generation using LLM-Codec and LLAMA2 model.The influence of multi-scale RVQ We first conduct experiments to see the effectiveness of multiscale RVQ. As Table 6 shows, compared with vanilla RVQ, the proposed multi-scale RVQ does not bring a significant reconstruction performance drop, which validates our assumption that semantic information does not need too much token to encode. Secondly, we find the multi-scale RVQ significantly reduces the length of the token sequence and brings benefits for downstream tasks (the audio classification accuracy is better than the baseline). One potential explanation is that LLMs can better identify the unique pattern in a brief sequence. Intuitively, the semantic information included in a 1-second audio is limited. It is unnecessary to use very long sequences to represent limited information.The influence of down-sampling times We can see that using a smaller down-sampling rate (320) can improve the reconstruction performance, but it also increases the length of the token sequence. We can see that the classification accuracy will decrease when the sequence length increases.The influence of semantic loss Without semantic loss, the performance of the audio understanding task will drop. Furthermore, we also find that adding semantic loss does not influence the reconstruction performance. In summary, the proposed semantic loss is very useful.The influence of consistency loss We find that consistency loss is important to maintain training stability. Without it, we can see the model fails to reconstruct the audio. We conjecture that frozen codebooks and large compression rates significantly improve the difficulty of training. The consistency loss forces the second VQ layer to produce features similar to those of the Whisper encoder, which provides guidance for vector quantization and prevents the model from collapsing in the early stage.The influence of word-level codebooks We also conduct experiments to show the effectiveness of using word-level codebooks to initialize the first VQ layer. Compared with using sub-word vocabulary for the first VQ layer, we can see that using the proposed word-level codebook can improve the reconstruction performance and classification accuracy.The importance of frozen codebooks LLM-Codec compresses the audio data into the token space of LLMs by initializing the codebooks with the LLMs' vocabulary and fixing it during the training stage. Table 6 also presents the results of updating codebooks: it can improve the reconstruction performance, but the accuracy is a significant drop. The result is consistent with our hypothesis: updating the codebooks parameter will decrease the codec training difficulty, but the learned codebook space is different from the LLM's token space, resulting in the downstream task performance declines. Different setting of k 1 and k 2 in multi-scale RVQ We validate a new setting for multi-scale RVQ with k 1 = 3 and k 2 = 5. We can see that the reconstruction performance will decline. We think one of the reasons is that the second VQ layer should not apply a large down-sampling step, which significantly influences the reconstruction.Codebook usage Previous works [18, 52, 25] suggest that using a large-scale codebook may result in codebook collapse ((where a fraction of the codes are unused). We calculate the codebook usage for each VQ layer in LLM-Codec. The used codes are 3246 (3248), 31911 (32000), 31941 (32000) for each VQ layer, which shows that most of codes are used. We present two groups of samples, each containing two audio recordings with the same sound event label. In each group, we use the same color to highlight potentially similar patterns in the two audio recordings, such as identical token sub-sequences or token repeating frequencies. We speculate that these patterns can be easily recognized by LLMs, allowing them to learn new sound events quickly with just a few demonstrations. 
conclusion: 
 In this study, we explore a cross-modal in-context learning approach to solve unseen audio tasks in a few-shot style. Specifically, we propose to train a LLMs-driven audio codec (LLM-Codec) that compresses the audio signal into the token space of LLMs. The LLM-Codec effectively reduces the modal heterogeneity between text and audio. With the help of LLM-Codec, pre-trained LLMs can be applied to solve multiple audio understanding and generation tasks. We demonstrate that LLM-Codec has good reconstruction performance, and the compressed token sequence is suitable for LLMs to understand and generate. Experiments show that the LLMs equipped with the proposed LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, are capable of achieving the expected functions in many scenarios.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section B in Appendix with data and model configurations information. We also report hyper-parameters of LLM-Codec.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitations: 
 Although we show the possibility of using the in-context learning ability of LLMs for unseen audio tasks without any parameter update, the performance of these tasks is still poorer than these special models in the audio domain. The capability to learn within an in-context framework is significantly limited for a modality that was not exposed during the training process. Due to the LLM's context length limitation, we cannot add more demonstration samples to help improve the performance. We think it is worth exploring using more demonstrations to improve its in-context learning ability. Moreover, we only explore the use of the LLAMA 2 7B model as the backbone; more advanced opensourced LLMs are worth exploring. Considering we have open-sourced the LLM-Codec, readers can conduct experiments on their favorite LLMs. Furthermore, there are fewer theoretical connections to justify the meaning of the lexical representation of the "trainable new (pseudo) language" of speech, which can be improved. In future work, we will explore to train multi-modal LLMs by fine-tuning LLMs on text-audio datasets with the help of LLM-Codec, and build more theoretical analysis.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report the mean value for experiments to run 5 times.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Section F in Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: These Appendices provide additional details to support our main manuscript, including ( 1 Encoder and Decoder Considering a single-channel audio signal x ∈ R t×sr , where t and sr denote the audio duration and the sample rate. The overall architecture is similar to previous audio codec models, such as Encodec [9] , DAC [25] , and HiFi-Codec [47] , which includes four main parts: encoder, quantizer, decoder, and discriminators. Figure 2 provides a visual depiction of the proposed method. For any input x, the encoder first encodes it into latent presentations E T,d , where T denotes the number of frames, d denotes the dimension of each vector. Due to the encoder includes some down-sampling layers, resulting in T << t × sr. Then each frame e ∈ E is passed through the quantizer, which assigns it to the closest entry in a codebook, resulting in the quantized embedding ê. Finally, the quantized feature Ê inputs into the decoder to reconstruct x. The encoder and decoder architecture follows previous works Encodec [9] and DAC-Codec [25] , which includes several convolution layers and transformer layers. Specifically, the encoder model comprises a 1D convolution with C channels and a kernel size of 7, leading into B convolution blocks. Each block contains a residual unit followed by a down-sampling layer, which employs a convolution with a kernel size K that is twice the stride S. The residual unit itself comprises two convolutions, each with a kernel size of 3, linked by a skip connection. The transformer block is used for sequence modeling, and concludes with a final 1D convolution layer featuring a kernel size of 7. In this study, we set S = [3, 4, 5, 8] , which results in 480 times down-sampling for audio. The decoder mirrors the encoder's architecture, substituting stride convolutions with transposed convolutions and reversing the stride order.Discriminators For the discriminators, we follow previous work [48] , which combines the mel-spectrogram and log-mel-spectrogram features and then input them into a network consisting of several convolutional layers. In our experiments, we use 6 different discriminators with different configurations. Specifically, we set the hidden dimension as {64, 128, 256, 512, 512, 512} and the hop length as {32, 64, 128, 256, 512, 1024}.The reconstruction loss is calculated between x and x. We design the loss from two aspects: the time domain and the frequency domain. For the time domain, we directly calculate the L 1 loss between x and x. For the frequency domain, we calculate the L 1 loss between the STFT spectrogram of x and x. Note that a sub-band split strategy [43] is used to split the spectrogram into several parts, and then we calculate the loss between these sub-bands. The adversarial loss is used to improve the perceptual quality of generated audio. A multi-scale Mel-spectrogram discriminators [48] is used. To train the discriminator, we can optimize the following objective function:EQUATIONwhere K denotes the number of discriminators. In the training stage, the adversarial loss for the generator is calculated as a hinge loss over the logits of these discriminators:EQUATIONWe also compute the feature loss by taking the average absolute difference between the discriminator's internal layer outputs for the generated audio and those for the corresponding real audio.The AdamW optimizer is used in the training. We set the learn rate as 1e -4. We train the model with 100k steps. For the training loss, we combine all of the loss terms without a special loss design.In the training stage, we use the pre-trained T5-base model and Whisper-base model for the reason that their latent dimension is both 512. We conduct all of the experiments with 2 NVIDIA A100-80G GPUs.In this part, we show how to construct an evaluation dataset for the N-way-k-shot test.Speech emotion classification with LLAMA 2. We give an example of 2-way 1-shot classification tasks. Firstly, we get the emotion class set from the ESD dataset: ['Angry', 'Happy', 'Neutral', 'Sad', 'Surprise']. Then we randomly choose two emotions as targets, and get the corresponding audios. For example, assuming that we get Happy and Sad the prompt can be For each of the following input-output pairs, the output is one of ['Happy' or 'Sad'] ### Input: <token sequence from a happy emotion of audio> Output: happy ### Input: <token sequence from a sad emotion of audio> Output: sad ### Input: <token sequence from the query audio> Output:We use greedy decoding to get a maximum of 16 tokens from LLAMA 2 7B.Sound event classification with LLAMA 2. We give an example of 3-way 1-shot classification tasks. Firstly, we get the sound event class set from the ESC50 dataset. Then we randomly choose three sound events as targets, and get the corresponding audio. For example, assuming that we get dog, speaking, and mouse click we set the prompt as To simplify to generation process, we set each audio has the same duration.Text-to-speech question design we designed 20 different questions for text-to-speech, which include addition, subtraction, multiplication, division, and reasoning.### Input: <an audio of (1+1)> ### Input: <an audio of (1+2)> ### Input: <an audio of (2+2)> ### Input: <an audio of (5-1)> ### Input: <an audio of (5-2)> ### Input: <an audio of (1-1)> ### Input: <an audio of (0 * 2)> ### Input: <an audio of (2 * 2)> ### Input: <an audio of (1/1)> ### Input: <an audio of (2/1)>Query: an audio of (dog bark) Query: an audio of (mouse click) Figure 6 shows the details of three VQ layers token visualization. We have the following findings: (1) The few tokens in the first layers seem to more easily understand the audio's pattern. For example, we can easily find two audios with the same sound event that can be quantized into very similar sequences. Because we force the first VQ layer to encode the semantic level information. Instead, the second and third VQ layers aim to encode the acoustic information, but these audios have obvious differences in acoustic condition (we can observe it from its mel-spectrogram). Second, it is worth noting that all of the training data is English-related, but we can see that the encoded sequence also includes other languages, such as Chinese.E Additional experiments in the rebuttal period E.1 Is there any scaling effects of the backbone LM selection?Inspired by the reviewer's suggestion, we added an experiment to explore the influence of scaling effects of the backbone LM. Specifically, we compare the performance of different LM selections: LLAMA2 7B and LLAMA 2 13B. We conduct experiments on N-way-1-shot sound event classification, The performance comparison as Table 9 shows. We can see that scaling the backbone LM can also bring improvement for audio tasks. We add a text-to-audio evaluation. Specifically, we choose previous SOTA AudioGen [24] as one of the baselines, because AudioGen is also an autoregression model based on audio codec models. Furthermore, we also choose some diffusion-based audio generation models, including AudioLDM and Tango, as the other baselines. For AudioLDM and Tango, we use their official checkpoints, and we set 200 diffusion steps for the inference. We conduct experiments on the ESC 50 [29] validation set. AudioGen, AudioLDM, Tango, and our model do not see the ESC 50 dataset in the training stage. We use the event label to construct the text description, e.g. if the event label is 'clapping', we will construct the caption as 'this is the sound of clapping.'. For the evaluation metrics, we follow previous works to use FAD and KL as the metrics. The results are shown in the table 10 . This paper aims to build a multi-modal LLM for audio understanding and generation tasks in a few-shot style without any parameter update for LLMs, which will ease the effort to develop different special models for different tasks. A negative impact is the risk of misinformation. To alleviate it, we can train an additional classifier to discriminate the fakes. We believe the benefits outweigh the downsides. The proposed method lowers the requirements for designing many special models, which may cause unemployment for people with related occupations, such as audio engineers.Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.https://github.com/yangdongchao/LLM-Codec 


Decision: The paper presents LLM-Codec, an innovative audio codec model that utilizes Large Language Models (LLMs) to handle various audio tasks with minimal training data. By converting audio signals into the token space of LLMs, the model enables these language models to comprehend and generate audio content.

Four reviewers assessed this work, and there was a good discussion period. The authors have sorted out several key concerns raised by the independent reviewers, and some reviewers have modified their overall evaluation of the paper. Unfortunately, two key issues still remain unsolved, and both are acknowledge by the authors during the discussion phase, namely: (i) poor presentation -  it needs improvement, and (ii) current model's performance is worse than a common baseline, AudioLDM -  the authors' claim should be then tuned down. Nonetheless, all reviewers agree that the topic is interesting and the idea can be considered novel.

Reviews: review: paper_summary: paper_summary: The authors introduce a three-step audio-to-discrete codec to encode continuous acoustic information into a form suitable for large language model-based audio and speech understanding.

Overall, this method is novel and represents an important step in audio modeling. The architecture targets different levels of acoustic information, from semantics to acoustic representation, using only discrete codecs. The results are empirically strong and solid.

However, there are fewer theoretical connections to justify the meaning of the lexical representation of the "trainable new (pseudo) language" of speech, which can be improved in future work.

For example, former works on word-level model reprogramming, learning equivalent pseudo tokens [A] from random embeddings, and theoretical bounds [B] on connecting different layers for latent alignment (e.g., when and how to align these three RVQ adapters) could help justify the uniqueness of the latent distance compared to embedding injection-based methods. This analysis could strengthen the theoretical foundations of the paper.

Despite these points, the overall contributions remain high-quality. 

- A few grammatical and formatting issues need to be very fixed in the final draft for a much ready version. 

In sum, I highly recommend accepting this paper and suggest the authors address these issues in the final version.
strength_weakness: strength_weakness: 1. there are less discussion on the representation difference of proposed codec-based method to the embedding injection based works. 

2. some minor grammar and formation issues

3. there are less theoretical discussion on the representation related to how many RVQ layers needed eventually
questions: questions: See the weakness (to be improved)

1. Are there any streaming or token merging limitation? 

2. Is there any scaling effects of the backbone LM selection?

3. what would be the semantic alignment from the codec to the lexical level representations?
limitations: limitations: minor, in terms of performance, the gap between cascaded LM for ASR and translation tasks.
paper_summary: paper_summary: The paper introduces LLM-Codec, a novel audio codec model that leverages Large Language Models (LLMs) to perform various audio tasks with minimal training examples. 

By translating audio signals into the token space of LLMs, it enables these models to understand and generate audio content. 

The model uses a multi-scale residual vector quantization approach to maintain audio quality while reducing token sequence length.
strength_weakness: strength_weakness: Experiments. 

Considering the paper proposes an encodec model, the most important result is the reconstruction performance. Providing Table 1, the necessary explaination of the results is lacking. I suggest this paper adds additional demonstration to the experimental main results to improve the readability.

Although the paper does a lot of experiments, showing promising results, the audio generation-related tasks still lack enough experiment results. In both introduction and related work section, the paper claims that previous codec models do not support  audio generation tasks, while the text-to-audio evaluation results are not shown. Considering the strong claim and the AudioCaps training data, it is strongly recommended to show its performance comapred to other text-to-audio models.
questions: questions: N/A
limitations: limitations: This paper provides sufficient discussion in this field.
paper_summary: paper_summary: The paper introduces LLM-Codec, which enables frozen LLMs to perform various audio tasks in a few-shot manner without fine-tuning LLMs. LLM-Codec operates in a RVQ-manner, hierarchically converting audio tokens into words or sub-words in the LLM vocabulary to compress the audio modality into the text space.
strength_weakness: strength_weakness: It’s hard to understand the meaning of the model setup. The authors train the system with distillation from T5 and Whisper instead of mapping them to the text-LLM like BLIP does. What is the motivation for building a system in this pipeline? Why is the decoder needed? What is the advantage of this approach compared to using external TTS or speech enhancement modules? These questions are difficult to answer with the current version of the presentation and authors should compare with BLIP-like approach to show its benefits.

From a presentation perspective, there are many grammatical errors and misleading sentences throughout the manuscript. The authors should put more effort into clarifying their claims and making the manuscript easier to comprehend by providing solid details.
- The captions of Figures 1 and 2 are hard to understand and contain grammatical errors.
- The details of the experimental setup are also insufficient to fully understand the setting.
- The purpose of Figure 4 is very unclear. There is no apparent correlation between the outputs of the semantic layer and the given audio. The purpose of this analysis is not evident, and the information provided is incomplete.

The authors conducted various downstream tasks, yet, the setup of experiments are insufficient to validate the system’s capacity. I’ve pointed out questionable setups at the Questions section.
questions: questions: - Why is RVQ setting adopted as the system pipeline? Have the authors try comparing with just performing downstream tasks using the outputs from T5 and Whisper?
- for Table 1,
    - What is the evaluation dataset being used? Does it only contain speech data?
    - How about performance comparisons with other metrics such as me reconstruction loss or SI-SDR?
    - How about using better configurations of the baseline models? (e.g., 44K DAC)
    - I didn’t fully get what 3 Vanilla RVQ means for Encodec_24k and DAC_16k.
- for Table 3, why is accuracy the only metric? How about other metrics such as AUC?
- for Table 4,
    - How is ACC being computed? How about WER?
    - What is the intuition of proposed method having better DNSMOS than the GT? Why not compute on subjective MOS?
limitations: limitations: Authors included their potential limitations in the Appendix section.
paper_summary: paper_summary: The paper proposes a LLM-codec module that can plug into an existing LLM, i.e., LLAMA-2, to perform few-shot in-context learning for tasks including classification (emotion & sound event), and text-to-speech synthesis. The proposed module takes the raw audio waveform as an input, and encodes it into latent space where the corresponding features are mapped into VQ codebooks in the LLM dictionary space. A multi-layer alignment designs are considered in the LLM-codec, where the shallow layer is responsible for semantic information, and deeper layers are responsible for more fine-grained information. Four losses are consider in aligning the features with the LLM pretrained embedding space, semantic loss, consistency loss, reconstruction loss and discriminator loss. For semantic loss and consistency loss, the goal is to guide the learned embeddings to have semantic meaning as well as align with the pretrained audio features for stability reason. Experiments demonstrate that the proposed module can be plugged into the pretrained LLAMA-2 for in-context few-shot learning for simple audio understanding tasks and TTS task.
strength_weakness: strength_weakness: 1. The task seems to be really simple, I am wondering how could the model performs under the more challenging scenarios as N-way goes larger? Also how the TTS performance degrades as the scripts become more complicated?

2. In table 2, for 2-way speech emotion classification, why is random guess only 40% accuracy rather than a number close to 50%. 59% acc is also not high enough for binary classification task. What is missing here in order to achieve better acc?

3. In Table 4, what is ACC for GT and FastSpeech 2, is the number not be able to compute?
questions: questions: See weaknesses above.
limitations: limitations: N/A

Comments: comment: Dear Reviewer oXws,

Thank you again for your tremendous efforts and valuable comments. We sincerely appreciate your recognition of our contributions and your constructive feedback. 

We agree that the current model’s performance in the text-to-audio task is still below that of previous specialized models, such as AudioGen and AudioLDM. We believe one reason for this is the difference in data coverage. In our study, we utilized only the AudioCaps dataset, whereas other specialized models have leveraged more extensive data sources, such as AudioSet. Therefore, a potential direction for improvement would be to scale up the data coverage.

Once again, we greatly appreciate that you recognize our contributions.  We sincerely thank you for your time, effort, and patience during this peer review process. We are always happy to have a further discussion and answer more questions raised by you.
Dear Reviewer 4YYB,

Thank you again for your tremendous efforts and valuable comments. We sincerely appreciate your recognition of our contributions and your constructive feedback. We are committed to improving our presentation based on your comments and suggestions.

We are currently revising the manuscript according to your feedback.  We hope this will address your concerns. We will continue refining our presentation in the coming days to ensure a polished final version.

In this version, we have mainly updated the following sections:

**1. We carefully reviewed our abstract and introduction to ensure proper grammar and enhance readability.**

**2. We updated the captions of Figures 1, 2, and 4. We hope the revised captions provide better clarity and address your concerns.** The details as follows:     
     **Figure1:** This figure illustrates the framework of the proposed approach for performing speech emotion classification and simple text-to-speech generation tasks. For each task, we prepare the instruction, demonstrations (e.g., x_1, y_1, x_2, y_2 ), and the query x_q. The LLAMA 2 model is then asked to predict the corresponding result y_q. Here, y_q can be either text or audio.

  **Figure 2:** This figure provides a high-level overview of LLM-Codec, including an encoder, a decoder, a multi-scale discriminator, and a multi-scale residual VQ layers. Here, ‘sub’ denotes feature subtraction. Note that the modules marked with a snowflake are frozen during training. 

**Figure 4:**  The token visualization of the semantic layer of LLM-Codec is shown. We present two groups of samples, each containing two audio recordings with the same sound event label. In each group, we use the same color to highlight potentially similar patterns in the two audio recordings, such as identical token sub-sequences or token repeating frequencies. We speculate that these patterns can be easily recognized by LLMs, allowing them to learn new sound events quickly with just a few demonstrations.

**We rewrote the Experimental Setting section to improve clarity. This section is now divided into two subsections: The first subsection provides detailed information about LLM-Codec, including the training data, codec model configuration, evaluation data, evaluation metrics, and the corresponding audio codec model baselines. The second subsection details the integration of LLM-Codec with pre-trained LLMs for downstream tasks (e.g., emotion classification, sound event classification, text-to-speech, etc.), including the evaluation data for each downstream task and the compared baselines..** 

Once again, we greatly appreciate that you raised the score and believe your valuable comments have significantly improved the paper, offering more precise explanations and presentations. We sincerely thank you for your time, effort, and patience during this peer review process.  We are always happy to have a further discussion and answer more questions raised by you.
Thanks for your explanation. I appreciate your efforts to add experiments comparing with the existing audio generation models. It is because that only when the related experiments are conducted, we can say if the paper overclaims its generation ability, which is states in the introduction section.

From the results, we can find that the current model's performance is worse than a common baseline, AudioLDM 1. Considering the results, I personally suggest that the paper may use the term "support audio generation tasks" more carefully. 

Hence, I will not improve or reduce the current score. Overall it is a very good paper discussing a useful topic.
I appreciate the authors' efforts in their rebuttal and have carefully read through all the reviews and additional clarifications provided. This has given me a clearer understanding of the paper's contributions.

However, my primary concern remains with the paper's presentation. The current version lacks clarity, which makes it difficult to fully appreciate the work's contributions. For this reason, I remain hesitant to recommend a solid acceptance. It is crucial that the final version significantly improves the clarity and presentation to effectively communicate the findings.

That said, I do acknowledge the interesting empirical findings presented within the existing pipeline, which has led me to slightly raise my score.
Dear Reviewer Gqom,

Thank you again for your great efforts and the valuable comments. Your suggestion significantly improves our work. We will add the extra discussion in the final version.
Thanks for the authors’ response. I think the original suggested references are missing due to some open review formatting. 

On the token level exploration, the most representative work is WRAP in ACL 2021 and the first speech model prompting work, Voice2Series in ICML 2021 has provided a general population risk bound for 1d vector discrete matching via measurement. 

The authors could strength their work with wider audiences based on more in-depth connections on these two well known works.

But I think the current version is acceptable for my evaluation for Neurips; although with relatively shallow theoretical findings. 

I recommend to accept this work and please add these extra discussion in a final version.
We thank the reviewers for the time and effort. We will further improve our presentation in the final version.
I have read through the authors’ response and it addresses all of my questions clearly by providing sufficient further experiment results. I keep my original rating and would lean towards the acceptance of the paper due to its novelty of design in LLM-Codec, albeit needing some improvement in presentation.