UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner
Authors: Dongchao Yang
Haohan Guo
Yuanyuan Wang
Rongjie Huang
Xiang Li
Xu Tan
Xixin Wu
Helen M. Meng
Venue: NeurIPS 2024 poster (2024)
Keywords: Audio undertanding and generation
in-context learning
Large language models
few-shot learning.
Primary area: speech_and_audio

Abstract: Large Language models (LLMs) have demonstrated supreme capabilities in textual understanding and generation, but cannot be directly applied to cross-modal tasks without fine-tuning. This paper proposes a cross-modal in-context learning approach, empowering the frozen LLMs to achieve multiple audio tasks in a few-shot style without any parameter update. 
Specifically, we propose a novel LLM-driven audio codec model, LLM-Codec, which transfers the audio modality into textual space by representing audio tokens with words or sub-words from the LLM vocabulary, while maintaining high audio reconstruction quality.
The key idea is to reduce the modality heterogeneity between text and audio by compressing the audio modality into the well-trained textual space of LLMs. Thus, the audio representation can be viewed as a new \textit{foreign language}, and LLMs can learn the new \textit{foreign language} with several demonstrations. In experiments, we investigate the performance of the proposed approach across multiple audio understanding and generation tasks, \textit{e.g.} speech emotion classification, audio classification, text-to-speech generation, speech enhancement, etc. Experimental results show that LLMs equipped with the LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, can perform effectively in simple scenarios, validating our cross-modal in-context learning approach.
To facilitate research on few-shot audio task learning and multi-modal LLMs, we have open-sourced the LLM-Codec model.

UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner
Dongchao Yang', Haohan Guo', Yuanyuan Wang?, Rongjie Huang', Xiang Li? Xu Tan Xixin Wu', Helen Meng'
1 The Chinese University of Hong Kong, 2 Tsinghua University, 3 Microsoft Research Asia (dcyang, hmmeng) ese. beuk. edu.hk
Abstract
Large Language models (LLMs) have demonstrated supreme capabilities in tex- tual understanding and generation, but cannot be directly applied to cross-modal tasks without fine-tuning. This paper proposes cross-modal in-context learn- ing approach, empowering the frozen LLMs to achieve multiple audio tasks in a few-shot style without any parameter update. Specifically, we propose a novel LLM--drvee audio codec model, L.MM-Codec, which transfers the audio modality into textual space by representing audio tokens with words or sub-words from the LLM vocabulary, while maintaining high audio reconstruction quality. The key idea is to reduce the modality heterogeneity between text and audio by compressing the audio modality into the well-trained textual space of LLMs. Thus, the audio representation can be viewed as new foreign language, and LLMs can learn the new foreign language with several demonstrations. In experiments, we investigate the performance of the proposed approach across multiple audio understanding and generation tasks, e.g. speech emotion classification, audio classification, text-to- speech generation, speech enhancement, etc. Experimental results show that L.LMs equipped with the I.LM-Coder named UniAudio 1.5, prompted by only few examples, can perform effectively in simple scenarios, validating our cross-modal in-context learning approach. To facilitate research on few-shot audio task learning and multi-modal LLMs, we have open-sourced the LLM-Codec model.
1 Introduction
Large language models (LLMs) (e.g., GPT-4 22. LLAMA (36) have become increasingly versatile and effective in handling diverse and complex Natural Language Processing (NLP) tasks as they scale in model size and training data. is worth noting that the in-context learning ability of LLMs can be used to solve unseen tasks, e.g., can provide instructions along with few demonstrations, enabling LLMS to learn and solve new tasks. The success of LLMs inspires the development of multi-modal LLMs, naturally leading to the idea of empowering their auditory capabilities to tackle audio-related tasks. There have been notable advancements in extending the capabilities of LLMs to tackle audio understanding tasks by combining the pre-trained audio encoder (e.g. Whisper encoder (3I) and LLMs. For instance, models like WavLLM (T5), SALMONN [35), and Qwen-audio [8] propose training multi-modal LLMs by integrating pre- trained audio encoder, trainable adaptor, and pre-trained LLMs. They try to align the audio and text modalities by updating the adaptor or fine-tuning the LLMs with LORA [4]. However, previous works have limitations: (1) they primarily focus on expanding L.MSs to solve specific audio tasks without leveraging in-context learning for
unseen audio tasks; (2) they do not support audio generation tasks, which limits their applicability; (3) they require large-scale audio data for aligning audio and text modalities, increasing the burden of model training and data collection.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Cedle- deceder
[Hppyy]
Sequensc
LLAMA
Word Sequence
Sequence
Wood Sequesce
Word Sequetice
the following ipptt
forcign language for differeni digit, then
LLo-Coode
M-CoCo
L.MM-Codee
J.MoCoadee
TEMSCaOe
generate corresponding number using forcign
output poins, is enc of emotions:
[add
[Happy]
langunge bisod instruction
[Sad or Happy]
audio
An audio of
audio
3/2
221
222
3/2
2y
(a) Speech emotion classification with in-context learming
(b) Text-to-speceh generation with in-context learning
This figure illustrates the framework of the proposed approach (UniAudio 1.5) for per- forming speech emotion classification and simple text-to-speech generation tasks. For each task, we prepare the instruction, demonstrations (e.g., {21, y1, 42, (2)), and the query Zig. The LLAMA model is then asked to predict the corresponding result Dg- Here, M/ can he either text or audio.
Figure 1:
In this study, we propose a cross-modal in-context learning approach, empowering the frozen LLMs solve any user-defined audio tasks based on few demonstrations without any parameter update. To achieve this, we introduce a vector quantization audio codec model, named LLM-Codec, that maps the audio modality the token space frozen LLMs (e.g.. LLAMA 2 [36.) Our motivation is to reduce modality heterogeneity between audio and text by compressing audio data into the token space of LLMs. Given that the compressed audio and text modalities share a vocabulary, the compressed audio sequence can be treated as new foreign language, which LLMs can learn from a few demonstration samples. Moreover, since LLMs are pre-trained on large-scale data and have discovered numerous token sequence patterns, they are well-positioned to generalize to this new foreign language. Figure Iillustrates the integration of the proposed LLM-Codec with LLAMA models for performing various audio tasks.
The proposed LLM-Codec aims to compress audio data into lexical word sequence. A desired LLM-Codec should exhibit the following properties: (1) Completeness [12]: it should recover compressed audio with minimal loss. (2) Compactness: it should encode the audio into fewer-token sequences. (3) Semantic richness: it should encode audio into semantically rich token sequences, making them easier for pre-trained LLMS to recognize. Thus, we propose a semantic- guided multi-scale residual vector quantization (RVQ) based codec. Specifically, the codec model consists of three residual VQ layers: the first layer encodes semantic information, the second encodes coarse-grained acoustic information, and the third encodes residual acoustic information. Unlike previous works (2), which encode audio data into the same granularity in each layer, we propose multi-scale approach that encodes audio data at different granularities across layers. We are motivated by the observation that semantic-level information can preserved with fewer tokens, while acoustic-level information requires more tokens This multi-scale approach not only shortens the token sequence length but also offers flexibility for various tasks; for instance, audio understanding tasks may only require the semantic-level VQ layer. Additionally, we design novel semantic loss and consistency loss to enhance the training of the L.LM-Cdde model.
We conduct experiments validate the effectiveness of ALM--odee in in-context learning setting. Using the pre- pre-trained LLAMA 2 7B model without parameter updates, we evaluate LLM-Codec on various audio understanding and generation tasks, such as speech emotion classification, audio classification, simple text-to-speech, and speech denoising. The main contributions of this work are summarized follows:
We propose a novel LLMs driven audio codec model, LLM-Codec, which effectively bridges the text and audio modalities. To the best of our knowledge, this is the first work to quantize the audio data into the representation space of L.LMs.
We demonstrate the feasibility and potential of using the in-context learning ability of LLMSs to solve unseen audio tasks, including audio understanding and generation tasks. Extensive experiments and ablation studies further validate the effectiveness of our method.

---

2 Related works
Audio Codec Models Audio codec models have been widely used in the audio generation domain [50, 48
46 Historical investigations into low-bitrate parametric audio codecs began with earlier studies 21 4 However, the quality of these codecs typically faced limitations. Recently, advancements in neural network-basse audio codecs have led to several promising developments [52 9 47 25 20. These systems typically involve encoder that extracts deep features from a latent space, which are then quantized and transmitted to a decoder for reconstruction. Particularly relevant to our work are the FACodec [20] and SpeechTokenize
(54) models, which explicitly model different properties of audio in different vector quantization layers. In contrast, our proposed LLM-Codec encodes audio data into lexical word sequence, differing from these approaches.
Multimodal Large Language Models Recently, there has been tremendous progress in the area of multimodal LLMs. These models use pre-trained LLMs as the base and incorporate additional input modalities, such as vision [55] 28 51 56 37 and audio 23 15, 53 35 19] [4]. In general, multimodal LLMs consist of a pre-trained LLM, pre-trained vision/audio encoder, and modality adaptor. These systems often require the construction of extensive multimodal datasets to fine-tune the models. In the audio modality, most previous works focus solving speech understanding [15] or general audio understanding (35 19,, but these models are not applicable to audio generation tasks. SpeechGPT addresses some audio understanding and generation tasks by fine-tuning all parameters and expanding the speech token vocabulary based on LLAMA. However, the speech tokens in SpeechGPT contain only semantic-level information, limiting its application to broader audio tasks, such as text-to-audio generation. Additionally, SpeechGPT does not explore the in-context learning ability to handle unseen tasks.
In-context Learning In-context learning, a form of few-shot learning, allows a large language model (LLM) to quickly adapt to a specific task during inference by reviewing just few examples provided in the prompt (6) This approach has proven successful in both natural language [45] and
disual-language tasks B 51 56 28 In the audio domain, several advanced methods have been proposed to leverage in-context learning for solving unseen audio tasks. SALM 7 introduces speech- augmented language models using in-context learning to address speech recognition and translation tasks, demonstrating that the SALM model can also handle keyword boosting. ICL-GSLM [13] employs warmup training and prompt tuning strategies to enhance the in-context learning abilities of pre-trained speech language models [26] for unseen tasks. However, ICL-GSLM primarily focuses on audio understanding tasks and overlooks audio generation tasks. Dynamic-supert [16] employs instruction tuning for audio understanding tasks. Similarly, [42] and [1 investigate in-context learning within the speech understanding domain. Building on the success of in-context learning in NLP 45) and vision-languag tasks [5] [6]] 28], our study focuses on harnessing the in-context capabilities of frozen LLMs address a broad range of audio understanding and generation tasks.
LLM-Codec
3.1
Overview
Previous audio codec models 9,52 47
use a VQ-VAE 38 framework, where the audio signal is first encoded into a discrete latent space and then decoded back into audio. Because audio codec models map the audio signal into discrete token sequences, many studies 17
48 5
have proposed
training auto-regressive (AR) language models to generate these sequences, inspired by the success of L.LMs in natural language processing. However, there is modal heterogeneity between the discrete audio tokens produced by the codee models and the text tokens used in LLMs. For example, the codebooks in audio codecs and the vocabularies of LLMs are typically unconnected, making it difficult to extend well-trained LLMs audio modalities. Although previous works [53 34 have demonstrated the effectiveness of expanding the vocabulary of LLMs to audio tokens and updating all of the parameters of LLMs, will cost a lot of computing resources and forget the knowledge of audio codec model (LLM-Codecc)
the text. In this part, we present a large language models which effectively bridges the gap between audio and text modalities. LLM-Codec is built on the VQ- VAE framework, but it differs from previous work in several key ways: (1) LLM-Codec is forced to quantize the audio signal into the token space of L.MMs; (2) L.MM-oddec adopts a multi-scale residual vector quantization strategy to balance the completeness and compactness of codec model;
3

---

wod-level Fedkbookk
MQ.2 sebwoiilevee audchenik
3 wewwand-eerd cedebook
Consatencc Losk
Lass
15 & Whisper
1234
6
6
Al inicer Layer
Bs
Lince Layerr
quartic
Up- sumpling
Up- sumpling
Down sampling
B
Bg Bg + Bg
8
Down sampling
Decoder
Multi scale Disctimitatie
Figure This figure provides a high-level overview of LLM-Codee, including an encoder, decoder, a multi-scale discriminator, and multi-scale residual VQ layers. Here, 'sub" denotes feature subtraction. Note that the modules marked with snowflake are frozen during training.
(3) LLM-Codec explicitly encodes different level information different VQ layers. The following sections provide detailed insights into LLM-Coder with Figure offering visual depiction.
3.2 Encoder and Decoder
For any audio æ, the encoder first encodes into latent presentations ET.d, where denotes the number of frames, d denotes the dimension of each vector. We set 4 down-sampling layers with 4, (.5.8) 8] in the encoder, which results in 480 times down-sampling for audio. Then, each frame E E is passed through the quantizer, which assigns it to the closest entry in codebook, resulting in the quantized embedding è. Finally, the quantized feature È inputs into the decoder to reconstruct a. Note that we add several Transformer layers in both Encoder and Decoder part to maintain good reconstruction performance. Refer to Appendix B to find more model structure details.
3.3 Multi-scale residual vector quantization with the vocabulary of frozen LLM
We use three residual VQ layers to maintain the balance between completeness and compactness. Furthermore, we propose to set different quantization granularity in different VQ layers: we expect the first VQ layer can encode the semantic information, and such information can be saved with fewer tokens, thus an interpolation function is used down-sample the encoder features TT, into
ET//k.dd then T//kid is passed through the first VQ layer obtain E T/ki,d For the second VQ layer, we expect it can encode coarse carse-grained acoustic information, thus we pass the residual of the
T.d first VQ layer into the next VQ layer. Before that, we first up-sampling E into E, then obtain the residual features by
T''ki,d
E Td
- E
(1)
Similarly, we also apply down-sampling operation to F..d we set the down-sampling step as k2-
T/k2.dd The features become as ET /k2,d Then we pass it into the second VQ layer and obtain E Lastly, we expect the last VQ layer can preserve all of the residual acoustic information. We first obtain the residual features based on the quantized features of the first two VQ layers
=
T.d T.d E,
(2)
Considering the residual acoustic information more complex and diverse, we directly apply the VQ operation to each frame without any down-sampling. By using large down-sampling step in encoder codec, and applying a multi-scale VQ strategy, we can effectively reduce the number of quantized audio token sequences. In our setting, 1-second audio with a 16k sampling rate will be quantized into 57 tokens. To ensure that the first VQ layers encode semantic information, we propose incorporating semantic loss during the training process. Furthermore, maintain the

---

Table 1: Performance comparison between open-sourced audio codec models, baselines, and the
proposed LLM-Codec. means the reproduced results by ourselves.
Model
Down-sampling
Tokens per second
PESQ
STOI
SFTF loss
Encodec_ 24k (3 Vanilla RVQ) 9
320
225
2.18
0.79
1.21
DAC 16k (3 Vanilla RVQ) 25
320
150
1.76
0.78
1.43
Baseline* (3 Vanilla RVQ)
480
99
2.64
0.83
1.09
Baseline* (2 Multi-scale RVQ)
480
41
2.22
0.79
1.23
Baseline* Vanilla VQ)
480
33
2.01
0.76
1.26
LLM-Codec (Ours)
480
57
2.55
0.82
1.15
training stability, we propose consistency loss. The details will be introduced in Section 3.4 The initialization of VQ layers To generate lexical tokens, we utilize a pre-trained LLAMA 2 codebook initialize the VQ layers. Considering that the first layer. the VQ layer, is designed to encode the semantic information, we do not directly use the full LLAMA codebook. Instead, we define new codebook based on Oxford 5000 Words, these words are commonly used to make up any meaningful sentence. We choose these words that only consist of one two sub-words in the LLAAMA codebook. a word includes two sub-words, we use the mean representation of two sub- words in the LLAMA codebook as the final representation. Lastly, the codebook size of the first VQ layer is 3248. We directly use the L.AAMA codebook to initialize the second and third VQ layers. The codebook size is 32000. Furthermore, the LLAMA codebook embedding dimension is 4096, which is too large for codec training. Thus, we apply a linear mapping to 512. In the training process, the parameters of codebooks are fixed.
3.4 Training loss
Our approach is based on GAN objective, in which we optimize both the generator(it consists of encoder, quantizer, and decoder) and the discriminators. For the generator, its training loss consists of three parts: (1) reconstruction loss term; (2) adversarial loss term (via discriminators); and (3) semantic and consistency losses. In the following, we give the details of proposed semantic loss and consistency loss. Refer to Appendix B.2 to find the details of reconstruction loss and adversarial loss.
Semantic loss To enhance the semantic representation ability in the first layer, we introduce a semantic loss for the first VQ layer. We expect it can encode semantic information, for example, if the input audio includes a sound event, the first layer should encode which semantic information of the sound event. Similarly, if the input audio is speech, the first layer should encode the content of the speech. To realize this target, we use a pre-trained T5-base model (32 to extract global representation vector g for the input audio content. We use Whisper to obtain its transcriptions if the input audio is speech. the input audio sound, we use its audio caption label:
C. = Li (mean BTT") ).g)
Consistency loss In our early experiments, we found the training of LLM-Codee is not stable, and the model easy to collapse. One of the reasons that we designed a significant down-sampling rate and the codebooks are fixed in the training, which increases the training difficulty. To solve this issue, we propose consistency loss maintain the training stability. Specifically, we propose using pre-trained Whisper encoder (31 to extract frame-level features then using these features as prior knowledge to guide the second VQ layer.
C. LE T//2. ,inp(w))
where inp denotes the interpolation function to align the feature dimension between the quantized features and whisper features. We chose the Whisper encoder because is trained not only on speech data but also on non-speech data. Furthermore, we do not apply this loss on the third VQ layer, because we expect the third VQ layer to encode the residual information.
4 UniAudio 1.5
By combining the pre-trained L.MSs and the proposed I.LM-Codec models, we can solve many audio tasks in few-shot style, as Figure shows. We named the system UniAudio 1.5 for the reason that
5

---

Table 2: Audio understanding task evaluation results. Task induction denotes the explanatory text that precedes the sequence of audio and text. is intended to describe the task to the model natural language, for example: Please answer the question. Accuracy (%) ) is used as the metric. For the Random guess, we calculate the average based 5 times evaluation. K shots refers to the number of distinct samples for each category, and Repeats refer to how many times we copy the prompt samples.
Task Induction K Shots Repeats
X
4
v
v
v
Method
Layers
0
0
0
2-way speech emotion classification
Random
None
44
BLSP 40
Whisper encoder
9
29
50
33
19
LLM-Codec
semantic layer
25
53
59
53
54
LLM-Codec
semantic + acoustic layers
45
49
53
55
54
2-way sound event classification.
Random
None
45
BLSP 40
Whisper encoder
44
47
semantic layer
48
60
57
57
73
LLM-Coder semantic+acoustic layers
41
48
55
3-way sound event classification.
Random
None
30
BLSP 440]
Whisper encoder
23
26
36
24
16
LLM-Codec
semantic layer semantic+acoustie layers
38
41
39
43
42
25
37
35
44
50
the system can be viewed universal audio task solver.
Connection UniAudio UniAudio 1.5 is an advanced edition of the UniAudio Series [48]. Com- pared to its previous version UniAudio 4 UniAudio 1.5 has the following connections and distinctions. First, goal. While both UniAudio and UniAudio 1.5 aim at building a universal audio foundation model for all audio tasks, their focuses different. UniAudio focuses audio generation tasks, such as text-to-speech, text-to-music, singing voice generation, and so on. UniAudio 1.5 focuses audio understanding and generation tasks by exploring the few-shot ability based on large language models. Second, architecture. UniAudio 1.5 keeps the basic components in UniAudio, such and a decoder-only transformer backbone used. However, UniAudio 1.5 leverages 1) a pre-trained LLMs solve the audio understanding and generation tasks by in-context learning, 2) LLM--diven audio codec to quantize the audio data into the token space of LLMs. Building a multi-modal audio foundation model that capable of handling any audio task the ultimate goal of the UniAudio series. In UniAudio 1.0, we show the possibility of building a universal model for different types of audio generation tasks, but it (1) cannot effectively solve audio understanding tasks and (2) cannot solve unseen audio tasks in the training or fine-ture fine-tuning stages. UniAudio 1.5 shows the possibility of using pre-trained L.LMs for both audio understanding and generation tasks. We believe the proposed L.LM-Codec in UniAudio 1.5 builds foundation for more advanced editions of the UniAudio Series the future.
audio codec used transfer the audio modality into discrete representations,
Experimental Results
5.1 Experimental Settings
5.1.1 LLM-Codec training and reconstruction performance evaluation
Training data LLM-Codec is a universal audio codec model, trained on both speech and sound datasets. For speech data, we use a portion of the MLS dataset [30], and for sound data, we use the AudioCaps dataset [22]. In total, we utilized 2k hours of audio data to train the LLM-Codee model.
Model setting The details of the LLM-Codec model configuration can be found in Appendix B For
proposed multi-scale RVQ, we set three scales, with down-sampling rates of k = 4, ho2 2, and kg for each layer. We initialize the VQ layers using the vocabulary of the LLAMA 2 7B model.

---

Table 3: Evaluation dynamic-superb benchmark tasks. Accuracy (%) is used as the metric.
Task
ImageBind-LLLMM
[6
Whisper-LL.M 16
ASR-ChaGGPT
Ours
Accent Classification
19
Bird Sound Detection
28
14
15
50
Chord Classification
44
58
55
Laagguage Identification
26
13
96
25
The VQ layers are fixed during the training stage.
Evaluation metrics To verify the reconstruction performance, we use Perceptual Evaluation of Speech Quality (PESQ), Short-Tima Objective Intelligibility (STOI), and Mel reconstruction loss. Evaluation data We conduct evaluation speech and sound datasets, for speech data, choose 200 utterances from VCTK [39], and for sound data, we choose 200 utterances from ESC dataset. Baselines We compare model with publicly available audio codec models, including Encodec [ and DAC [25]. Additionally, we compare with our reproduced audio codec models, which were trained on the same dataset LLM-Codec.
5.1.2 LLMs equipped with the proposed LLM-Codec for downstream audio tasks
Evaluation dataset We choose commonly used test datasets for each task and construct N- N-way-K- shot test pairs. More details about constructing evaluation samples can be found Appendix [.T Baselines Given the limited number of works focusing few-shot learning for unseen audio tasks, we choose BLSP 44) as one of the baselines for audio understanding tasks. Since BLSP is fine-tuned on continuation writing task and does not explicitly address audio classification, these tasks considered unseen for the BLSP model. Additionally, we compare our approach with instruction- tuning- based models as described in cmnamic-ssuete [16]. For audio generation tasks, did not find directly related works, so we report the performance of state-of-the-art specialized models.
5.2 Main results
We first present the reconstruction performance comparison. Then we apply the LLM-Codec and LLAMA 7B model for audio understanding and audio generation tasks, to verify the ability of the proposed method. Lastly, we give the visualization of LLM-Codec to explain why it can work. We leave more experiments on Appendix
Reconstruction performance We compare the audio reconstruction quality with previous works Encodee (2) DAC-Codec (25), and our baseline model. We report Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibilityy (STOI). Table [ shows the re- sults. Compared to previous methods, the L.M-Codec achieves better reconstruction performance while utilizing fewer tokens. More specifically, the LLM-Codec model can compress 1-second audio data into a sequence that only includes 57 tokens, which significantly reduces the sequence length. Compared to the RVQ baseline model, the LLM-Codec significantly reduces the com- pressed tokens, and its reconstruction performance does not significantly decline. In Section [5.3 will show the importance of compressing audio into fewer tokens. We also conduct experi- ments to validate whether we can use a few VQ layers, such as 1 VQ layer or 2 VQ layer, we can see that the reconstruction performance will significantly drop. To maintain the balance be- tween completeness and compactness, we choose a multi-scale 3 VQ layer as the default setting. Speech Emotion Classification The speech emotion classification task [10] aims to predict the emotion label of the speech. We conduct 2-
Table 4: Text-to-speech generation performance.
way K-shot experiments on the ESD [1] dataset.
Experimental results are shown in Table 2
We
Model
ACC
DNSMOS
have the following findings: (1) Task induction important to maintain the stability of perfor- mance, we can see that without task induction, the classification accuracy will dramatically de- cline. (2) The semantic layer effectively extracts the global semantics of audio, which can be easily understood by the LLAMM model. (3) Using more demonstration samples (e.g. 3 shots), the performance will be better. (4) Repeating the demonstration
GT
2.91
FastSpeech 2
3.42
LLM-Codec (Ours)
70
2.92
7

---

samples can also bring improvement. (5) Compared to the BLSP, our method performs better in any setting. Furthermore, we also note that the performance of BLSP will drop when repeat operation is used. One possible reason is that BLSP only learns the translation relationship between text and speech, repeating samples cannot bring new cues for L.MMs to solve the new task. Instead, our LLM-Codec learns to map the audio data into the latent space of L.MMs, increasing the number of demonstration samples can help LLMs to find special patterns to solve this new task.
Sound Event Classification Sound event classification aims to recognize the sound event in the audio. In general, an audio may include multiple events. To simplify the recognition difficulty, we assume each audio only includes one event. We conduct experiments on the ESC50 dataset [29]. which includes 50 different types of events. We construct 2- 2-way-K-shh K-shot and 3-way-K --sott evaluations based on the ESC50 test set. Compared with the BLSP model, our proposed method gets better performance. Based on the experimental results from two audio understanding tasks, we can see that the semantic VQ layer is very important for understanding tasks.
Dynamic-SUPERRB Benchmark We also conduct experiments on Dynamic- SUPERB Benchmark tasks [6]. In [16), authors propose instruction-tuning strategy for multi-modal LLMs. They first construct a lot of audio tasks as training data, then validate some unseen audio tasks in a zero- shot way. To make fair comparison, we use the same test set with them, and choose the first N samples as the demonstration to construct a N- N-way-1-sher audio understanding tasks, our proposed method obtains better or compared performance than these baselines in [6]. Especially, for the bird sound detection task, our proposed method obtained great improvement over previous methods. We also note that our method performs worse on language identification, the possible reason is that our codec model is only trained on English speech data. In the following, we will show that L.M-Codec also can be used to conduct audio generation tasks.
evaluation. As Table shows 4 selected
Simple texe-to-speech generation We conduct text-to-speech generation on the Free Spoken Digit Dataset (FSDD) dataset W. which includes 3 speakers and 1,500 recordings. Unlike the traditional TTS model, which generates any speech content, this task generates digit speech. Our current model to generate complex speech content is still challenging. We use accuracy (ACC) to assess the content of the generated sample whether following the instructions. DNSMOS is used to assess the speech quality of generated samples. We construct 20 different query questions, including addition, subtraction, multiplication, division, and reasoning (finding more details from Appendix[C..
From Table 4 we can see that our proposed model can accurately understand the query in most cases (the accuracy is 70 %) and generate good-quality speech samples. Figure gives a visualization of generating speech based on the query. The frozen LLAMÁ model learns about 4 digits (0-3), each audio digit includes 5 samples. We add the context for each audio: "an audio of k" before inputting the audio's discrete representations into LLAMA, as Figure (b) shows. After that, we let the LLAMA 2 model generate corresponding speech digits based on the instruction. We also note that the generated audio appears different from all context audio samples, demonstrating the cross- modal reasoning capability of LLMs when using the LLM-Codes as the connector for text and audio.
Simple speech denoising To verify whether the pro- posed method can conduct speech-denoising tasks, we simulate noisy speech based on the VCTK dataset and NoiseX-92 dataset, we set the SNR ranges from -20 to 20. For each clean speech, we choose 5 different noises to simulate noisy speech. The first 4 noisy and clean audio pairs are used as demonstrations, and the model learns to denoise the last noisy one. To improve in-context learning ability, we repeat the demonstration samples 4 times. The experimental results as T we can see that the proposed method can also learn to denoise without any training. Furthermore, we also note that the performance has a large room to improve compared to special models.
Table 5: Speech denosing evaluation.
Model
PESQ
STOI
SGMSE+ 33
3.53
0.79
LLM- LLM--Cddc (Ours)
2.17
0.57
Token Visualization We visualize the tokens produced by the first VQ layer of LLM-Codec for different types of sound in Figure 4 We have the following findings: (1) Although the two audios include the same sound event, the quantized sequence is not exactly the same. (2) The quantized
sequence of two same types of audio has similar pattern, e.g. their token sequences have similar repeating patterns or the same word. Such patterns may help the LLMS recognize the type of audio.
8

---

Pronup
Query: an audio of (0-2)
Quary: an audio of (1+1)
Query: audio of (1-1)
o-2-0
1+1-2
Figure 3: Examples of simple text-to-speech generation using LLM-Codec and LLAMA2 model.
5.3
Ablation study
The influence of multi-scale RVQ We first conduct experiments to see the effectiveness of multi- scale RVQ. As Table 6 shows, compared with vanilla RVQ. the proposed multi-scale RVQ does not bring a significant reconstruction performance drop, which validates assumption that semantic information does not need too much token to encode. Secondly, we find the multi-scale RVQ sig- nificantly reduces the length of the token sequence and brings benefits for downstream tasks (the audio classification accuracy is better than the baseline). One potential explanation is that LLMs can better identify the unique pattern in brief sequence. Intuitively, the semantic information included in a I-second audio is limited. is unnecessary to use very long sequences to represent limited information.
The influence of down-sampling times We can see that using a smaller down-sampling rate (320) can improve the reconstruction performance, but also increases the length of the token sequence. We can see that the classification accuracy will decrease when the sequence length increases. The influence of semantic loss Without semantic loss, the performance of the audio understanding task will drop. Furthermore, we also find that adding semantic loss does not influence the reconstruc- tion performance. In summary, the proposed semantic loss very useful.
The influence of consistency loss We find that consistency loss is important to maintain training stability. Without it, we can see the model fails to reconstruct the audio. We conjecture that frozen codebooks and large compression rates significantly improve the difficulty of training. The consis- tency loss forces the second VQ layer to produce features similar to those of the Whisper encoder, which provides guidance for vector quantization and prevents the model from collapsing in the carly stage.
The influence of word-level codebooks We also conduct experiments to show the effectiveness of using word-level codebooks to initialize the first VQ layer. Compared with using sub-word vocabulary for the first VQ layer, reconstruction performance and classification accuracy.
see that using the proposed word-level codebook can improve the
The importance of frozen codebooks LLM-Codec compresses the audio data into the token space of LLMs by initializing the codebooks with the LLMs' vocabulary and fixing it during the training stage. Table also presents the results of updating codebooks: it improve the reconstruction performance, but the accuracy is significant drop. The result consistent with our hypothesis: updating the codebooks parameter will decrease the codec training difficulty, but the learned codebook space is different from the LLM's 's token space, resulting in the downstream task performance declines. Different setting of K; and k22 in multi-scale RVQ We validate a new setting for multi-scale RVQ with k 3 and kn 5. We can see that the reconstruction performance will decline. We think one of the reasons is that the second VQ layer should not apply a large down-sane sampling step, which significantly influences the reconstruction.
Codebook usage Previous works I8 52 25 suggest that using a large-scale codebook may result in codebook collapse ((where a fraction of the codes are unused). We calculate the codebook usage for each VQ layer in L.MM-Codec. The used codes are 3246 (3248), 31911 (32000), 31941 (32000) for each VQ layer, which shows that most of codes are used.

---

Table 6: Ablation studies training loss, multi-scal RVQ setting, initialization of VQ layer. The
classification accuracy (%) )) is evaluated under the sound event classification task 2-way 1-shot setup.
Model
Down-sampling
Tokens second
PESQ
STOI
ACC
Baseline (3 Vanilla RVQ)
480
99
2.64
0.83
55
LLM-Codec Multi-scale RVQ)
480
57
2.55
0.82
60
LLM-Codec Multi-scale RVQ)
320
2.60
0.83
57
w/o semantic loss
480
57
2.54
0.82
58
consistency loss
0.53
48
w/o word-level codebook
480
57
2.46
0.81
59
updating codebooks
480
57
2.63
0.83
55
seting 11 and l2
480
50
2.35
0.79
58
Erpsstops
Brihsteps
scoring
spoing
getrogan
Semvatic
Dendome Uherefoe uolunty mubody swilch Uhreten binkeming conoent Uhrefom
phnas phnas
Lay
thersfore concoming Booods moboay disk
yesterday yesterlay whatever pewpppaee Sunday
planse after phraze tham
The token visualization of the semantic layer of LLM--odec shown. We present two groups of samples, each containing audio recordings with the same sound group, we use the same color to highlight potentially similar patterns in the two audio recordings, such identical token sub-sequences or token repeating frequencies. We speculate that these patterns can be easily recognized by LLMs, allowing them to learn new sound events quickly with just few demonstrations.
Figure 4:
label. In each
6 Conclusion
In this study, we explore moosl in-context learning approach to solve unseen audio tasks in few-shot style. Specifically, we propose train LLMs-driven audio codec (LLM-Codecc that compresses the audio signal into the token space of LLMs. The LLM-Codec effectively reduces the modal heterogeneity between text and audio. With the help L.M--odeec, pre-trained LLMss can be applied to solve multiple audio understanding and generation tasks. We demonstrate that LLM-Code has good reconstruction performance, and the compressed token sequence suitable for LLMs to understand and generate. Experiments show that the LLMs equipped with the proposed LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, are capable achieving the expected functions in many scenarios.
7 Limitations
Although show the possibility of using the in-context learning ability of LLMs for unseen audio tasks without any parameter update, the performance of these tasks is still poorer than these special models in the audio domain. The capability learn within in-context framework significantly limited for a modality that was not exposed during the training process. Due to the LLM's context length limitation,
cannot add more demonstration samples help improve the performance. We think it is worth exploring using more demonstrations to improve its in-context learning ability. Moreover, we only explore the use of the LLAMA 7B model the backbone; more advanced open- sourced LLMs are worth exploring. Considering we have open-sourced the LLM-Codec, readers can conduct experiments on their favorite L.MMs. Furthermore, there are fewer theoretical connections to justify the meaning of the lexical representation of the "trainable new (pseudo) language" of speech, which can be improved. In future work, we will explore to train multi-modal L.LMs by fine-tuning LLMs on text-audio datasets with the help of LLM-Codec, and build more theoretical analysis.
10

---

8 Acknowledgements
This work is partially supported by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd., a CUHK-led under the InnoHK scheme of Innovation.
References
[1] Emotional voice conversion: Theory, databases and esd. Speech Communication, 137:1-18. 2022.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736. 2022.
[4] Bishnu S Atal and Suzanne L Hanauer. Speech analysis and synthesis by linear prediction of the speech wave. The journal of the acoustical society of America, 50(2B):637-655, 1971.
[5] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 13:187711 2020.
[7] Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. Salm: Speech-augmentted language model with in in-context learning for speech recognition and translation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13521-13525. IEEE, 2024.
[8] Yunfci Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07915 2023.
[9] Alexandre Défosse7, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint 1X.::22/0..34333 2022.
[10] Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern recognition, 44(3):572-553 -87, 2011.
[11] Zohar JacksonCésar. etall. free- sneessooen-ddgii-daaasse htt.aeenooo..44444.. 1342401, 2018.
v1.0.8 (v1.0.8). zenodo.
[12] Haohan Guo, Fenglong Xie, Xixin Wu, Frank K. Soong, and Helen Meng. Msmc-tts: Multi- stage multi-codebook vq-vae based neural tts. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1811-1824 2023.
[13] Ming-Hao Hsu, Kai-Wei Chang, Shang-Wes Wen Li, and Hung-yi Lee, An exploration of in-context learning for speech language model. arXiv preprint arXiv:2310. /2477, 2023.
[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09680, 2021.
[15] Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint 3XXv:2404.00655 2024.

---

[16] Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi- Yuan Hsiao, Chun- Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-superb: Towards dynamic, collaborative, and comprehensive ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12136-122 12140.. IEEE, 2024.
paaruction-tuning benchmark for speech. In
[17] Rongjie Huang, Chunlei Zhang. Yongqi Wang, Dongchao Yang. Luping Liu, Zhenhui Ye, Ziyue Jiang, Chao Weng, Zhou Zhao, and Dong Yu. Make-a-voice: Unified voice synthesis with discrete representation. arXiv preprint arXiv:2305. /9266, 2023.
[18] Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges in vector quantized networks. In International Conference on Machine Learning. PMLR, 2023.
[19] Atin Sakkeer Hussain, Shansong Liu, Chenshuo Sun, and Ying Shan. Mugen: Audti-modal music understanding and generation with the power of large language models. arViv preprint arXiv:2311.11255 2023.
[20] Zeqian Ju. Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint Skdv::2403.03100 2024.
[21] Biin--Hwang Juang and A Gray. Multiple stage vector quantization for speech coding. In ICASSP'82. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 7, pages 597-600. IEEE, 1982.
[22] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 119-132, 2019.
[23] Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint urXiv:2402.01833 2024.
[24] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint 220..:52229915555... 2022.
(25) Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
[26] Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, AbdeIrahman Mohamed, al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:133--1354, 2021.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[28] Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards un- supervised text-image alignment. Advances in Neural Information Processing Systems, 36. 2024.
[29] Karol Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pages 1015-1018. ACM Press.
[30] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: large-scale multilingual dataset for speech research. arXiv preprint 2:X1v:2012.03411,2222 2020.
12

---

[31] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Machine Learning, pages 28492-28518. 2. PMLR, 2023.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-55 2020.
[33] Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. JEEE/ACM Trunsuctions on Audio, Speech, and Language Processing, 31 31:2351-2364, 2023.
[34] Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia EI Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.1292. 2023.
[35] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310. 13289, 2023.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[37] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212 2220002-122000 2021.
[38] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[39] Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit. 2017.
[40] Chen Wang, Minpeng Liao, Zhonggiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. Blsp: Bootstrapping language-specce pre-training via behavior alignment of continuation writing, arXiv preprint arXiv:2309.009) 2023.
[41] Siyin Wang, Chao-Har Yang. Ji Wu, and Chao Zhang. Can whisper perform speech- speech-based based in-context learning? In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13421-1344 13425. IEEE, 2024.
[42] Siyin Wang, Chao-Han Huck Yang, Ji Wu, and Chao Zhang. Bayesian example selec- tion improves in-context learning for speech, text, and visual modalities. arXiv preprint a404 /4716, 2024.
[43] Yuanyuan Wang. Hangting Chen, Dongchao Yang, Jianwei Yu, Chao Weng. Zhiyong Wu, and Helen Meng. Consistent and relevant: Rethink the query embedding in general sound separation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 961-965. IEEE, 2024.
[44] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 1884 03209, 2018.
[45] Jason Wei, Maarten Bosma, Vincent Y Zhao. Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Lee Finetuned language models are zero-shot learners. arXiv preprint 2rXiv:2109.01665: 2021.
[46] Donghaa Yang, Rongjie Huang, Yuanyuan Wang, Haohan Guo, Dading Chong, Songxiang Liu, Xixin Wu, and Helen Meng. Simplespeech 2: Towards simple and efficient text-ho-speech with flow-based scalar latent transformer diffusion models. arXiv preprint arXiv:2408. 13893,
2024.
13

---

[47] Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codes Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765, 2023.
[48] Dongchao Yang. Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: Towards universal audio generation with large language models. ICML 2024, 2023.
[49] Dongchao Yang, Dingdong Wang, Haohan Guo, Xueyuan Chen, Xixin Wu, and Helen Meng. Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models. InterSpeech. 2024, 2024.
[50] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. /EEE/ACM Transactions on Audio, Speech, and Language Processing. 2023.
[51] Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang. David Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen Ilms. Advances in Neural Information Processing Systems, 36, 2024.
[52] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507 2021.
[53] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang. Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305. 11000, 2023.
[54] Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. /6692, 2023.
[55] Kaizhi Zheng. Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and iision-and-languee lan-uuge generation via generative vokens. arXiv preprint arXiv:2310.02239, 2023.
[56] Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond text: Frozen large language models visual signal comprehension. arXiv preprint 07X7722403.08874, 2024.
14

---

Appendices
A Appendix Overview
These Appendices provide additional details to support our main manuscript, including (1) the training detail and model structure of ALd-C..ee. (2) The details of the evaluation dataset. (3) More audio task evaluation results. (4) Limitations.
B
More details of LLM-Codec
B.1
Model structure
Table gives the details of LLM-Codec configuration, which results in 160M parameters. To facilitate research on cross-modal in-context learning and multi-modal LLMs, we have open-sourced the LLM-Codec models.
LLM-Codec
Input shape
(1, 1, T)
Encoder (input dimension)
32
Down-sampling rate
[3, 4, 5, 8]
latent dimension
512
Codebook dimension
4096
Transformer layer dimension
512
Number of Transformer heads
8 1536
Decoder dimension
Up-sampling rate
[8, 5, 4, 3] [5, 3, 1]
VQ strides
Table 7: LLM-Codec model backbone configurations
Encoder and Decoder Considering single-channel audio signal æ E RRtxer, where and sr denote the audio duration and the sample rate. The overall architecture is similar previous audio codec models, such as Encodec (9). DAC (25). and HiFi-Codec [47). which includes four main parts: encoder, quantizer, decoder, and discriminators. Figure2 method. For any input æ,, the encoder first encodes it into latent presentations E where T denotes the number of frames, d denotes the dimension of each vector. Due to the encoder includes some down-sampling layers, resulting in T << X sr. Then each frame E is passed through the quantizer, which assigns it to the closest entry in codebook, resulting in the quantized embedding è. Finally, the quantized feature É inputs into the decoder to reconstruct a. The encoder and decoder architecture follows previous works Encodec (9] and DAC-Codec 25 which includes several convolution layers and transformer layers. Specifically, the encoder model comprises a ID convolution with channels and a kernel size of 7, leading into convolution blocks. Each block contains a residual unit followed by down-sampling layer, which employs convolution with kernel size K that is twice the stride The residual unit itself comprises two convolutions, each with kernel size of 3, linked by a skip connection. The transformer block is used for sequence modeling, and concludes with final ID convolution layer featuring kernel size of 7. this study, set S [3, 4, 5, 8), which results in 480 times down- down-sampling for audio. The decoder mirrors the encoder's s architecture, substituting stride convolutions with transposed convolutions and reversing the stride order.
a visual depiction of the proposed
Discriminators For the discriminators. we follow previous work mel-spectrogram and log-mel- -egetsrspeetoo features and then input them into network consisting of several convolutional layers. our experiments, we use 6 different discriminators with different configurations. Specifically, we set the hidden dimension as (64, 128, 256, 512, 512, 512) and the hop length as [32, 64, 128, 256, 512, 1024).
which combines the
15

---

B.2 Reconstruction loss and adversarial loss for LLM-Codec
The reconstruction loss calculated between and x. We design the loss from two aspects: the time domain and the frequency domain. For the time domain, we directly calculate the L1 loss between æ and Φ. For the frequency domain, calculate the L1 loss between the STFT spectrogram of æ and at Note that sub-band split strategy 44 is used to split the spectrogram into several parts, and then we calculate the loss between these sub-bands. The adversarial loss used improve the perceptual quality of generated audio. A multi-scale Mel-spectrogram discriminators 48 is used. To train the discriminator, we can optimize the following objective function:
L Σ max (0, 1 Dx(x))) + max (0, 1 Dx()))
(5)
where K denotes the number of discriminators. In the training stage, the adversarial loss for the generator calculated as hinge loss over the logits of these discriminators:
K
Lade
Σ
mar (0, (x)
K
We also compute the feature loss by taking the average absolute difference between the discriminator's internal layer outputs for the generated audio and those for the corresponding real audio.
B.3 Training details
The AdamW optimizer is used in the training. We set the learn rate le 4. We train the model with 100k steps. For the training loss, we combine all of the loss terms without a special loss design. In the training stage, we use the pre-trained T5-base model and Whisper-base model for the reason that their latent dimension is both 512. We conduct all of the experiments with 2 NVIDIA A100-80G GPUs.
C Evaluation dataset
In this part, we show how to construct an evaluation dataset for the N-way-k-shot test.
C.1 N-way-k-shot test samples
Speech emotion classification with LLAMA 2.
We give example of 2-way 1-shot classification tasks. Firstly, we get the emotion class set from the ESD dataset: l'Angry', 'Happy', 'Neutral', 'Sad', 'Surprise']. ]. Then randomly choose emotions targets, and get the corresponding audios. For example, assuming that we get Happy and Sad the prompt can be
For each of the following input-output pairs, the output is one of ['Happy' or 'Sad')
###
Input: <token sequence from happy emotion of audio>
Output: happy 4#
Input: <token sequence from sad emotion audio>
Output: sad ###
Input: <token sequence from the query audio> Output:
We use greedy decoding to get maximum of 16 tokens from LLAMA 7B.
Sound event classification with LLAMA 2.
We give an example of 3-way 1-shot classification tasks. Firstly, we get the sound event class set from the ESC50 dataset. Then we randomly choose three sound events as targets, and get the corresponding audio. For example, assuming that we get dog, speaking, and mouse click we set the prompt
16

---

For each of the following input output pairs, output is one doo' or 'speaking' or mousee_click) ###
Input: <token sequence from dog event audio> Output: dog ###
Input: <token sequence from speaking event of audio> Output: speaking 4##
Input: <token sequence from mouse click event of audio>
Output: mouse click ###
Input: <token sequence from the query audio> Output:
C.2 Audio generation
Text-fo-speech generation with LLAMA 2
Learn a foreign language for different digits, then generate the corresponding number using foreign language based on instruction
Instruction:
4#
Input: <an audio of 1>
Output: <token sequence of audio l>
##
Input:
<an audio of 2>
Output: <token sequence of audio 2> ###
Input:
<an audio of 3>
Output: <token sequence of audio 3>
###
Input: <an audio of 1+1> Output:
To simplify to generation process, we set each audio has the same duration.
Text-to-speech question design designed 20 different questions for text-to-speech, which include addition, subtraction, multiplication, division, and reasoning.
##
Input:
<an audio of (1+1)>
##
Input:
<an audio of (1+2)> >
###
Input:
<an audio of (2+2)>
###
Input:
<an audio of (5-1)>
###
Inpuu:
<an audio of (5-2) >
##
Input:
<an audio of (1-1)
###
Input:
<an audio of (0+2)>
###
Input: <an audio of ) ###
Input: <an audio of (1/1)>
###
Input: <an audio of (2/1)>
17

---

4#
Input:
<an audio of (4/2)>
###
Input:
<an audio of (the square root of 4)>
###
Input:
<an audio of (the square root of 1)>
##
Input: <an audio of (the last digit of 110)>
###
Input:
<an audio of (the first digit of 110)>
##
Input:
<an audio of (the sum of 1+1+1)>
###
Input:
<an audio of (the next digit of 4)>
###
Input:
<an audio of (sequence 0,1,2,3 what is n
###
Input: <an audio of (sequence 4, 3, what is next?)> A##
Input:
<an audio of (how many days in a week)>
D More audio tasks evaluation experiments with the proposed method
In the following, we show the results of speech command recognition and text-to-sound generation.
D.1 Speech Command Recognition
Table 8: Speech command recognition evaluation results on Speech Command dataset. Accuracy (%) 5 times then calculate the average.
used the metric. For the Random guess,
Task Induction
X
v
Method
Layers
K Shots Repeats
0
3
LLM-Codec
semantic layer
50
53
59
54
56
LLMM-odee
semantic+acoustic layers
25
53
58
49
52
BLSP [40
Whisper encoder
29
65
84
69
59
Random
None
Speech command recognition refers to the recognition and interpretation short phrases or keywords that are typically used control devices applications. In this part, choose audio samples from the Speech Command dataset [44). We choose four types of commands, including down, go, left, and right. For each command, we randomly choose 20 utterances, then use these data construct 2- 2-way-K-sth evaluation. Experimental results are shown in Table 8 we can see that only using the semantic VQ layer brings the best performance. Instead, if the tokens from the acoustic layer used, the performance will decline. One possible reason is that for the speech command recognition task, only needs to understand the content, and the content information has been saved in the semantic layer, the additional acoustic information may disturb the LLMs's 's prediction.
D.2 Simple text-to-sound generation
Similarly, we can also use the same setting text-to-speech to conduct eoxt-to-sonnd generation tasks. We choose test set from the ESC50 dataset [29], and let the model learn to generate sound events based on the text label. For example, we can set several different sound types in the prompt, and then ask the I.LAMA model to generate a new audio. However, we also find that it hard ask LLM to generate new types of sound.
18

---

dog
mouse click
water drops
Prompt
Query: an audio of (dog bark)
Query: an audio of (mouse click)
dog
mouse
Figure 5: Examples of simple text-to-sound generation on FSDD dataset using L.M-Codec with
frozen LLAMA2 7B model.
footsteps
footstops
snoring
storing
Ulrtoore Unereforeooooo bobody Ulveden
therefore thersfor nobedy coneentngooooeeennn theefors
thernfore Bobodd
yestectig yestenda teaeppprr Sundyy
of plrpee
dsk
kvol Rain Cy Stants (I. Mai degree
Jabyre Onflicil istener Lord wodely eland Mechan onder lntial genecates parmeetrr Statls syea Alzhema serving states vause rolle Theme Indroct
éval Rain call plugins Hir 3 Poe
24th boyer
Varg dland spproatiedd tegp & Coass Number Barii and JSON lime Come Indenti
typeof Edmund mamgier
Beck strects
KirKs Gubill directory geotueatc rou stakerechngg TL.5 mise Come
subecttie Contei
Indesut
Soboction umop
turned ustraux Jt. Global Many wer pol bookcill Although Lip Britain nus James segment uncs zelle sow app
zumis Alt issues ond Il dumain Et CASS scler de encie links franptes liglish acton & rak arda inbegr di'
al scioni Browser Yagaz isted
dispute SWW qjoo samples Thess lsca uses mic pule es Hist ine constru sumss soll fortggaa zue Klam duse ply interaaional crved nsce Ruturo rex it aasod in Thorr
long Spect Here grondratisches tijd
difer Using Had servv pocially ença
34t VQ
pos lir FF 1ime was yaah men Footbell Marcos wicheelte numemus Rece
vent Toonlload aran PRF osyotol adlnn reset serverp Nedtng Ps.
dip volor Hi Far not ada wkk CSV
Pirrsorn boces wicht 7, une kh Thor
purch Jt Thor Estad supplied
of renwe Thar Estad supplie
Estas supplied
Estaal suplied
Figure 6: The token visualization of three VQ layers with LLM-Codec. The audio samples from the ESC50 dataset.
D.3 Token visualization
Figure shows the details of three VQ layers token visualization. We have the following findings: (1) The few tokens in the first layers seem to more easily understand the audio's 's pattern. For example, we can easily find two audios with the same sound event that can be quantized into very similar sequences. Because we force the first VQ layer to encode the semantic level information. Instead, the second and third VQ layers aim to encode the acoustic information, differences in acoustic condition (we can observe it from its mel-spectrogram) Second, it is worth noting that all of the training data English-related, but we can see that the encoded sequence also includes other languages, such as Chinese.
these audios have obvious
19

---

E Additional experiments in the rebuttal period
E.1 there any scaling effects of the backbone LM selection?
Inspired by the reviewer's 's suggestion, we added an experiment to explore the influence of scaling effects of the backbone LM. Specifically, we compare the performance of different LM selections: LLAMA2 7B and LLAMA 2 13B. We conduct experiments on N-way-I-shot sound event classifica- tion, The performance comparison as Table shows. We can see that scaling the backbone LM can also bring improvement for audio tasks.
Table 9: The influence of scaling effects the backbone LM.
Model task
2-way- I-soot
3-way- --sot
4-way- Issot
E-way-1-shon
6-way- I-soot
LLAMA 7B
60
41
36
33
17
LLAMA 13B
62
0.42
41
43
31
BLSP
47
26
15
12
10
E.2 The performance comapred to other text-to-audio models.
We add text-to-audio evaluation. Specifically, we choose previous SOTA AudioGen [24] as one of the baselines, because AudioGen is also an autoregression model based on audio codec models. Furthermore, we also choose some diffusion-based audio generation models, including AudioL and Tango, as the other baselines. For AudioL.DM and Tango, we use their official checkpoints, and set 200 diffusion steps for the inference. We conduct experiments set. AudioGen, AudioLDM, Tango, and our model do not see the ESC 50 dataset in the training stage. We use the event label to construct the text description, e.g. if the event label is 'clapping',
the ESC 50 [29] validation
will construct the caption as "this is the sound clapping.". For the evaluation metrics, we follow previous works to FAD and KL as the metries. The results are shown in the table I0
Table 10: The aux-to evaluation.
model
FAD
KL
AudioGin
20.4
1.94
AudioL
15.6
3.52
Tango
12.7
3.01
Ours
17.16
3.05
E.3 The reconstruction performance comparison ESC 50 dataset
Table 11: The reconstruction performance comparison. We compared with previous SOTA model, such as DAC-Codec and Encodec.
Model
PESQ
STOI
STFT loss
DAC (3 RVQ)
1.88
0.52
1.55
Encodec (3 RVQ)
2.01
0.54
1.39
ours
2.09
0.52
1.38
F Potential Negative Societal Impacts
This paper aims to build a multi-modal LLM for audio understanding and generation tasks few-shot style without any parameter update for LLMs, which will ease the effort to develop different special models for different tasks. A negative impact is the risk of misinformation. To alleviate it, we can train an additional classifier to discriminate the fakes. We believe the benefits outweigh the downsides. The proposed method lowers the requirements for designing many special models, which may cause unemployment for people with related occupations, such as audio engineers.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state the claims made, including the contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section ?? in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
21

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section Appendix with data, and model configurations information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.er cc/
public/ quwpic/cguee/eeeeeee CodeeSuio ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. //npsscc/pubbll wttpcccppppppppcteeeceecccccccep cesii///uies///eeeeeeeeeooeoseee ies CodeeSume ss cioooooliio ooooiccyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section ii Appendix with data and model configurations information. We also report
of LLM-Codee.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the mean value for experiments to run times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section in Appendix for information on compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:ttttssshuippsscooom http:///wrpppc.ccoooo/l///ccccce
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section F in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetttett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [TODO]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [TODO]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "qEk6gHMx4H", "reviewer": "Reviewer_Gqom", "summary": "The authors introduce a three-step audio-to-discrete codec to encode continuous acoustic information into a form suitable for large language model-based audio and speech understanding.\n\nOverall, this method is novel and represents an important step in audio modeling. The architecture targets different levels of acoustic information, from semantics to acoustic representation, using only discrete codecs. The results are empirically strong and solid.\n\nHowever, there are fewer theoretical connections to justify the meaning of the lexical representation of the \"trainable new (pseudo) language\" of speech, which can be improved in future work.\n\nFor example, former works on word-level model reprogramming, learning equivalent pseudo tokens [A] from random embeddings, and theoretical bounds [B] on connecting different layers for latent alignment (e.g., when and how to align these three RVQ adapters) could help justify the uniqueness of the latent distance compared to embedding injection-based methods. This analysis could strengthen the theoretical foundations of the paper.\n\nDespite these points, the overall contributions remain high-quality. \n\n- A few grammatical and formatting issues need to be very fixed in the final draft for a much ready version. \n\nIn sum, I highly recommend accepting this paper and suggest the authors address these issues in the final version.", "questions": "See the weakness (to be improved)\n\n1. Are there any streaming or token merging limitation? \n\n2. Is there any scaling effects of the backbone LM selection?\n\n3. what would be the semantic alignment from the codec to the lexical level representations?", "limitations": "minor, in terms of performance, the gap between cascaded LM for ASR and translation tasks.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. the model architecture design is overall new and effective \n\n2. the designs on the different concept of audio representation is interesting", "weaknesses": "1. there are less discussion on the representation difference of proposed codec-based method to the embedding injection based works. \n\n2. some minor grammar and formation issues\n\n3. there are less theoretical discussion on the representation related to how many RVQ layers needed eventually", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. We appreciate the constructive comments the reviewer provided to us to improve our paper further. We are delighted to have the following discussion with the reviewer.     \n**Q1:**  A few grammatical and formatting issues need to be very fixed in the final draft for a much-ready version.     \n**A:**  We appreciate this comment. We are glad to revise our paper for better writing clarity.     \n\n**Q2:** there are fewer theoretical connections to justify the meaning of the lexical representation of the \"trainable new (pseudo) language\" of speech, which can be improved in future work. For example, former works on word-level model reprogramming, learning equivalent pseudo tokens [1] from random embeddings and theoretical bounds [2] ...       \n**A:**  We appreciate and agree with the reviewer's suggestion. We also want to build a theoretical foundation for this study. We plan to refer to your mentioned reference paper pseudo tokens [1] and theoretical bounds [2], and try to learn how to build a theoretical foundation in the future. Can you give a detailed paper name or URL for [1] and [2]? We very much appreciate your help.\n\n**Q3:**   There is less discussion on the representation difference between the proposed codec-based method and the embedding injection-based works.    \n**A:**  We appreciate and agree with the reviewer's comment: we will add a discussion section to show the difference between our proposed codec-based method and previous embedding injection-based works, e.g. previous works (e.g. BLSP [1] or Qwen-Audio [2]) use Whisper model extracts continuous embedding, and then fine-tuning pre-trained LLM on audio understanding tasks. We will add the following content to show the difference:\n\n*(1) Motivation:* the embedding injection-based works expect that LLMs can understand audio embedding by fine-tuning LLMs so that their model supports input audio modality and output text content. Instead, our proposed method tries to transfer audio modality into LLM's token space, so that LLM can directly understand audio modality without additional fine-tuning.\n\n*(2) Formulation:* the embedding injection-based works need a fine-tuning LLMs stage. In general, they need to train an adaptor or fine-turn part of the parameter by the LORA strategy. Instead, our proposed method does not need any parameter updating for LLMs.\n\n*(3) Target:* most of the embedding injection-based works focus on audio understanding tasks. Instead, we expect to build a universal audio understanding and generation framework with the help of LLM-Codec.\n\n**Q4:**  there are less theoretical discussion on the representation related to how many RVQ layers needed eventually     \n**A:**  We appreciate and agree with the reviewer's comment. We are happy to discuss this issue: From a reconstruction performance perspective, using more RVQ layers will bring better performance. From a generation perspective, using more RVQ layers brings the long sequence problem for LM, furthermore, it also increases the inference costs. Thus, we seek a compact but complete audio representation, preserving sufficient semantic and acoustic information with few tokens. \n\n**Q5:**  Are there any streaming or token merging limitation?   \n**A:**   We are happy to discuss this issue with the reviewer. Previous codec models, such as Encodec and SoundStream, are both supporting streaming. One of the reasons is that these works adopt a causal convolution block in the Encoder and Decoder parts. In our codec, we also use similar convolution blocks with Encodec. So our codec also supports streaming. For the token merging limitation, we adopt the multi-scale RVQ strategy, which results in different VQ layers producing different numbers of tokens, which may bring challenges in token merging.\n\n**Q6:**   Is there any scaling effects of the backbone LM selection?      \n**A:**    We appreciate the reviewer's comment. Inspired by your suggestion, we added an experiment to explore the influence of scaling effects of the backbone LM. Specifically, we compare the performance of different LM selections: LLAMA2 7B and LLAMA 2 13B. We conduct experiments on N-way-1-shot sound event classification, The performance comparison as following Table shows. We can see that scaling the backbone LM can also bring improvement for audio tasks.  \n|   Model / task   | 2-way-1-shot  | 3-way-1-shot | 4-way-1shot | 5-way-1-shot | 6-way-1-shot |\n|:----------------:|:-------------:|:------------:|:-----------:|:------------:|:------------:|\n|  Ours (LLAMA 7B) |       60      |      41      |      36     |      33      |      17      |\n| Ours (LLAMA 13B) |       **62**      |     **42**     |      **41**     |      **43**      |      **31**      | \n\n**Q7:**  what would be the semantic alignment from the codec to the lexical level representations?   \n**A:**    We appreciate the reviewer's comment. In our view, the semantic token should have a strong connection or high correlation with the lexical-level representations. For instance, the same semantic information in two audios should be mapped into a similar lexical sequence, so that the LLMs can learn the the pattern with few demonstrations. \n\n[1] Wang C, Liao M, Huang Z, et al. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing[J]. arXiv preprint arXiv:2309.00916, 2023.\n\n[2] Chu Y, Xu J, Zhou X, et al. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models[J]. arXiv preprint arXiv:2311.07919, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVAFnIcy7V", "reviewer": "Reviewer_oXws", "summary": "The paper introduces LLM-Codec, a novel audio codec model that leverages Large Language Models (LLMs) to perform various audio tasks with minimal training examples. \n\nBy translating audio signals into the token space of LLMs, it enables these models to understand and generate audio content. \n\nThe model uses a multi-scale residual vector quantization approach to maintain audio quality while reducing token sequence length.", "questions": "N/A", "limitations": "This paper provides sufficient discussion in this field.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper shows a novel codec model for audio compression and can be generated by a frozen LLM for in-context learning.\n\n- The task setting is challenging, and the semantic codec is a challenging task, while the paper shows good exploration to it.\n\n- The ablation study is sufficient, showing the elements in the RVQ codec model's usages.", "weaknesses": "Experiments. \n\nConsidering the paper proposes an encodec model, the most important result is the reconstruction performance. Providing Table 1, the necessary explaination of the results is lacking. I suggest this paper adds additional demonstration to the experimental main results to improve the readability.\n\nAlthough the paper does a lot of experiments, showing promising results, the audio generation-related tasks still lack enough experiment results. In both introduction and related work section, the paper claims that previous codec models do not support  audio generation tasks, while the text-to-audio evaluation results are not shown. Considering the strong claim and the AudioCaps training data, it is strongly recommended to show its performance comapred to other text-to-audio models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. We appreciate the constructive comments the reviewer provided to us to improve our paper. We are delighted to have the following discussion with the reviewer.   \n**Q1:**  Considering the paper proposes an encodec model, the most important result is the reconstruction performance. Providing Table 1, the necessary explaination of the results is lacking. I suggest this paper adds additional demonstration to the experimental main results to improve the readability.  \n\n**A:**   We thank the reviewer for his/her valuable suggestions. We are glad to revise our paper to improve its readability. Specifically, we would: \n\n*(1)* Provide more explanation for reconstruction performance results in Section 4.2, including the reconstruction performance comparison, the influence of down-sampling steps and tokens per second. Furthermore, inspired by reviewer 4YYB, we add a new metric (the Mel reconstruction loss) into Table 1. \n\n*(2)* Highlight the advantages of our proposed codec. \n\n**Q2:** Considering the strong claim and the AudioCaps training data, it is strongly recommended to show its performance comapred to other text-to-audio models. \n\n**A:**   We appreciate this suggestion. We add a text-to-audio evaluation. Specifically, we choose previous SOTA AudioGen [1] as one of the baselines, because AudioGen is also an autoregression model based on audio codec models. Furthermore, we also choose some diffusion-based audio generation models, including AudioLDM [2] and Tango [3], as the other baselines. For AudioLDM and Tango, we use their official checkpoints, and we set 200 diffusion steps for the inference. We conduct experiments on the ESC 50 [4] validation set. AudioGen, AudioLDM, Tango, and our model do not see the ESC 50 dataset in the training stage. We use the event label to construct the text description, e.g. if the event label is 'clapping', we will construct the caption as 'this is the sound of clapping.'. For the evaluation metrics, we follow previous works to use FAD and KL as the metrics. The results are shown in the following table. Furthermore, we also add a visualization of generated samples in Figure 1. We will update this into our final version. \n\n|   model  |  FAD  |  KL  |\n|:--------:|:-----:|:----:|\n| AudioGen |  20.4 | **1.94** |\n| AudioLDM |  15.6 | 3.52 |\n|   Tango  |  **12.7** | 3.01 |\n|   ours   | 17.16 | 3.05 |\n\n[1] Kreuk F, Synnaeve G, Polyak A, et al. Audiogen: Textually guided audio generation[J]. ICLR 2023.\n\n[2] Liu H, Chen Z, Yuan Y, et al. Audioldm: Text-to-audio generation with latent diffusion models[J]. ICML, 2023.\n\n[3] Ghosal D, Majumder N, Mehrish A, et al. Text-to-audio generation using instruction-tuned llm and latent diffusion model[J]. ACM-MM, 2023.\n\n[4] Piczak K J. ESC: Dataset for environmental sound classification[C]//Proceedings of the 23rd ACM international conference on Multimedia. 2015: 1015-1018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vFnp3KI1FN", "reviewer": "Reviewer_4YYB", "summary": "The paper introduces LLM-Codec, which enables frozen LLMs to perform various audio tasks in a few-shot manner without fine-tuning LLMs. LLM-Codec operates in a RVQ-manner, hierarchically converting audio tokens into words or sub-words in the LLM vocabulary to compress the audio modality into the text space.", "questions": "- Why is RVQ setting adopted as the system pipeline? Have the authors try comparing with just performing downstream tasks using the outputs from T5 and Whisper?\n- for Table 1,\n    - What is the evaluation dataset being used? Does it only contain speech data?\n    - How about performance comparisons with other metrics such as me reconstruction loss or SI-SDR?\n    - How about using better configurations of the baseline models? (e.g., 44K DAC)\n    - I didn’t fully get what 3 Vanilla RVQ means for Encodec_24k and DAC_16k.\n- for Table 3, why is accuracy the only metric? How about other metrics such as AUC?\n- for Table 4,\n    - How is ACC being computed? How about WER?\n    - What is the intuition of proposed method having better DNSMOS than the GT? Why not compute on subjective MOS?", "limitations": "Authors included their potential limitations in the Appendix section.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 3, "strengths": "The approach is validated through experiments on tasks such as speech emotion classification, audio classification, text-to-speech generation, and speech enhancement, demonstrating its feasibility and effectiveness.", "weaknesses": "It’s hard to understand the meaning of the model setup. The authors train the system with distillation from T5 and Whisper instead of mapping them to the text-LLM like BLIP does. What is the motivation for building a system in this pipeline? Why is the decoder needed? What is the advantage of this approach compared to using external TTS or speech enhancement modules? These questions are difficult to answer with the current version of the presentation and authors should compare with BLIP-like approach to show its benefits.\n\nFrom a presentation perspective, there are many grammatical errors and misleading sentences throughout the manuscript. The authors should put more effort into clarifying their claims and making the manuscript easier to comprehend by providing solid details.\n- The captions of Figures 1 and 2 are hard to understand and contain grammatical errors.\n- The details of the experimental setup are also insufficient to fully understand the setting.\n- The purpose of Figure 4 is very unclear. There is no apparent correlation between the outputs of the semantic layer and the given audio. The purpose of this analysis is not evident, and the information provided is incomplete.\n\nThe authors conducted various downstream tasks, yet, the setup of experiments are insufficient to validate the system’s capacity. I’ve pointed out questionable setups at the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's time and patience with our paper. We do find these suggestions constructive and helpful. \n\n**Q1:** It’s hard to understand the meaning of the model setup ... What is the motivation for building a system in this pipeline?  \n\n**A:**  We apologize that our presentation cannot make the reviewer better understand our model setup and design. We are happy to discuss this with the reviewer. From the high-level, we aim to turn the powerful LLM into a universal audio understanding and generation model to tackle infinite audio tasks without training. Due to the length limitation,  we give the detailed explanation on global response part.\n\n**Q2:**  Why is the decoder needed? \n\n**A:**  Thanks for this comments. As we discussed, we expect the LLMs can generate audio directly. So we first use our LLM-Codec to quantize the audio data into the LLM's token space (word or sub-word). Then LLMs will predict the corresponding tokens based on the instruction, these tokens can be recovered into waveform by using the codec decoder. Such a strategy has been widely used in audio generation, such as AudioLM [5], researchers first use a codec model to compress the audio into discrete tokens, then use a transformer model to predict the corresponding tokens. Lastly, the codec decoder is used to recover waveform from the predicted tokens. \n\n**Q3:**  From a presentation perspective, there are many grammatical errors and misleading sentences throughout the manuscript. ....  \n\n**A:**  We appreciate this comment. We are glad to revise our paper for better writing clarity. \n\n**Q4:** Why is RVQ setting adopted as the system pipeline? \n\n**A:**  Thank you for this comment. The reason includes (1) we need to quantize audio into discrete tokens (2) RVQ is the mainstream approach for audio quantization. Furthermore, we use the LLM's vocabulary as the RVQ codebook, which means that we quantize the audio modality into the LLM's vocabulary. The benefits include: (1) with the help of LLM-Codec, LLMs can directly generate audio; (2) we do not need to fine-tune the LLMs. \n\n**Q5:**  Have the authors try comparing with just performing downstream tasks using the outputs from T5 and Whisper? \n\n**A:**  Yes, one of our baseline BLSP [1], which using a Whisper encoder to extract speech representation, and fine-tuning the LLM with a learnable adaptor. Due to BLSP only output text, we only compare with it on audio understanding tasks, such as speech emotion classification and sound event classification. \n\n**Q6:**  What is the evaluation dataset being used? Does it only contain speech data? \n\n**A:**  We appreciate this comment. As we introduced in Table 1 caption, we evaluate the reconstruction performance on the VCTK dataset. We randomly chose 200 utterances. Inspired by your suggestion, we also chose 200 utterances from the ESC50 dataset to evaluate the reconstruction performance on sound data. The results as Table 2 shows (refer to global response PDF).\n\n**Q7:**  How about performance comparisons with other metrics such as mel reconstruction loss or SI-SDR?\n\n**A:**  Based on your suggestion, we add the Mel reconstruction loss metric as one of the evaluation metrics. The performance as Table 3 shows. We will update this into our final version. \n\n**Q8:** How about using better configurations of the baseline models? (e.g., 44K DAC)  \n\n**A:**  We agree that the 44K DAC model has good reconstruction performance for high-sampling rate audio (e.g. 44.1k hz). However, in our study, we train all of the codec models on 16khz audio data. Thus, we choose 16K DAC-codec as one of the baselines. Due to the 16K Encodec model is not released, we downsampling the generated audio by Encodec_24k into 16k. \n\n**Q9:**  I didn’t fully get what 3 Vanilla RVQ means for Encodec_24k and DAC_16k.  \n\n**A:** We thank this important comment and appreciate the reviewer. In our study, we propose a multi-scale residual vector quantization, which is different from previous commonly used RVQ in DAC and Encodec, so we name previous RVQ as Vanilla RVQ. We apologize for this misunderstanding, we will add an explanation to the Table 1 caption part. \n\n**Q10:**  for Table 3, why is accuracy the only metric? How about other metrics such as AUC? \n\n**A:**  For the metric, we follow the baseline Dynamic-superb to use accuracy as the metric. We agree that AUC is a good metric to evaluate a classifier. We want to point out that AUC is calculated by setting different threshold values to get a group of FPR and TPR. However, we use the LLMs to directly predict the text label,  it is hard to set a 'threshold' like a traditional classification model. In other words, AUC is more suitable to evaluate the performance of traditional classifiers, in which humans can decide the threshold. If the reviewer can provide better metrics, we are happy to add them to the paper.\n\n**Q11:** for Table 4, How is ACC being computed? How about WER? \n\n**A:**  We calculate the ACC by comparing the content of the generated speech with the ground truth value. Actually, due to the generated speech only includes digits, using ACC and WER has the same meaning. \n\n**Q12:** What is the intuition of proposed method having better DNSMOS than the GT? Why not compute on subjective MOS? \n\n**A:**  The reason is that the GT from the Free Spoken Digit Dataset (FSDD) is low-quality. FSDD is an old speech digit dataset, which may include some noise. Instead, our audio codec model is trained on a clean and high-quality speech dataset, so the noise details will not be modeled by the codec model. Actually, such a phenomenon widely exists in speech generation, nowadays, many TTS models can synthesize better speech than the original one, e.g. NaturalSpeech 3 [8] shows it can generate better speech than LibriSpeech. Inspired by your valuable suggestion, we conducted a subjective evaluation as Table 4 shows.\n\nDue to the length limitation, we put the remaining question into global response part.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FlpQKx8Ku0", "reviewer": "Reviewer_XMxu", "summary": "The paper proposes a LLM-codec module that can plug into an existing LLM, i.e., LLAMA-2, to perform few-shot in-context learning for tasks including classification (emotion & sound event), and text-to-speech synthesis. The proposed module takes the raw audio waveform as an input, and encodes it into latent space where the corresponding features are mapped into VQ codebooks in the LLM dictionary space. A multi-layer alignment designs are considered in the LLM-codec, where the shallow layer is responsible for semantic information, and deeper layers are responsible for more fine-grained information. Four losses are consider in aligning the features with the LLM pretrained embedding space, semantic loss, consistency loss, reconstruction loss and discriminator loss. For semantic loss and consistency loss, the goal is to guide the learned embeddings to have semantic meaning as well as align with the pretrained audio features for stability reason. Experiments demonstrate that the proposed module can be plugged into the pretrained LLAMA-2 for in-context few-shot learning for simple audio understanding tasks and TTS task.", "questions": "See weaknesses above.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes an interesting way of solving few-shot audio-related tasks using frozen LLMs. Different from previous methods, this work designs a plug-in module for LLM in-context learning for audio modality to avoid LLM training or fine-tuning. The plug-in module is designed to be efficient, with only 160M parameters. \n\n2. The tasks can cover both audio understanding and simple text-to-speech synthesis, which is flexible enough considering the limitations of in-context learning.\n\n3. The paper is mostly well-written and presentation is clear enough to understand the motivation and the proposed method.", "weaknesses": "1. The task seems to be really simple, I am wondering how could the model performs under the more challenging scenarios as N-way goes larger? Also how the TTS performance degrades as the scripts become more complicated?\n\n2. In table 2, for 2-way speech emotion classification, why is random guess only 40% accuracy rather than a number close to 50%. 59% acc is also not high enough for binary classification task. What is missing here in order to achieve better acc?\n\n3. In Table 4, what is ACC for GT and FastSpeech 2, is the number not be able to compute?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. We appreciate the constructive comments the reviewer provided to us to improve our paper further. We are delighted to have the following discussion with the reviewer. \n\n**Q1:**  The task seems to be really simple, I am wondering how could the model performs under the more challenging scenarios as N-way goes larger?  \n\n**A:**  Thank you for your comments. We set different settings $N=2,3,4,5,6$ for the sound event classification task. The results are as follows. We can see that in the more complex scenarios, our proposed model can also get better performance than the baseline BLSP [1]. Furthermore, inspired by reviewer Gqom, we also find that using a larger LLM (e.g. LLAMA 2 13B) can further improve the performance.\n|   Model / task   | 2-way-1-shot  | 3-way-1-shot | 4-way-1shot | 5-way-1-shot | 6-way-1-shot |\n|:----------------:|:-------------:|:------------:|:-----------:|:------------:|:------------:|\n|  Ours (LLAMA 7B) |       60      |      41      |      36     |      33      |      17      |\n| Ours (LLAMA 13B) |       **62**      |     **42**     |      **41**     |      **43**      |      **31**      |\n|       BLSP       |       47      |      26      |      15     |      12      |      10      |\n\n**Q2:**  Also how the TTS performance degrades as the scripts become more complicated? \n\n**A:**  Thank you for your comment. Inspired by your suggestion, we set three difficulty levels to test the simple text-to-speech. The first level includes addition, subtraction, multiplication, and division, such as the speech of *2x2*. The second level includes simple reasoning, such as *what is the next for the sequence 0,1,2,3*. The third level includes complex reasoning, such as *if 2 times a number plus 1 equals 1, what is the number*. We ask ChatGPT to help construct the questions for each level. For each level, we test 20 utterances. As a result, we find that the accuracy for each level is 85\\%, 65\\%, and 60\\%. \n\n**Q3:**  In table 2, for 2-way speech emotion classification, why is random guess only 40 \\% accuracy rather than a number close to 50 \\%. \n\n**A:**  Thank you for your comment. As we introduced in the Table 2 caption, for the Random guess, we follow the previous work Dynamic-SUPERB [2], and we calculate the average score based 5 times evaluation. We agree with the reviewer's view that when we run the random guess enough times, the final accuracy will close to 50\\%. In order to give the reader a better understanding, we will update the random guess score as the theoretical probability.  \n\n**Q4:**  59\\% acc is also not high enough for binary classification task. What is missing here in order to achieve better acc? \n\n**A:**  Thank you for your comment. We have two reasons: 1) out-of-domain data leads to lower performance; 2) low understandablity of LLM to audio modality. We have following potential solutions to address these two issues: 1) providing high-diversity audio dataset for model training; 2) fine-tuning the LLM.\n\n**Q5:**  In Table 4, what is ACC for GT and FastSpeech 2, is the number not be able to compute?. \n\n**A:**  For FastSpeech 2, we directly input the ground truth answer text to the TTS model, so the generated speech is always right. When comparing with FastSpeech 2, we want to show that our proposed method can generate high-quality speech.\n\n[1] Wang C, Liao M, Huang Z, et al. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing[J]. arXiv preprint arXiv:2309.00916, 2023.\n\n[2] Huang C, Lu K H, Wang S H, et al. Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 12136-12140.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the meta-reviewer for organizing this helpful peer review stage. We thank all reviewers for their time, patience, and constructive comments to help us improve our paper. 

We specifically address the concerns raised by reviewers **4YYB** regarding the motivation of our paper. We are eager to engage in the following discussions:

**What is the motivation for building a system in this pipeline?** 

As we discussed in the Introduction part. The success of LLMs inspires many researchers to build multi-modal LLMs to solve audio-related tasks in the audio domain. For instance, BLSP [1], and Qwen-Audio [2] typically use a pre-trained Whisper encoder to extract speech embedding and then use a learnable adaptor module (e.g. Linear layer or Q-former [3]) and LORA to map the speech embedding into the representation space of LLM. Such a strategy has been widely discussed and recognized in the research community. However, previous works (1) focus more on expanding LLMs to solve specific audio tasks, without considering the in-context-learning ability to unseen audio tasks, e.g. Quen-Audio collects over 30 tasks to conduct multi-task training. Their motivation is using the strong ability of LLMs to help improve the performance of audio understanding tasks. (2) do not support audio generation tasks, which limits its application scenarios. In general, audio tasks can be divided into audio understanding and audio generation. We refer to the audio understanding task as the model's input can be text and audio, but the model's output is only text, e.g. Automatic speech recognition (ASR), Spoken language identification, Emotion recognition, and so on. Similarly, if the model's output is audio, we call these tasks as audio generation tasks, e.g. text-to-speech (TTS), text-to-sound, speech enhancement, and so on. We claim that  **one of our motivations is building a universal audio understanding and generation tasks solver with the help of LLMs and the proposed LLM-Codec**. We highlight that making LLMs to generate audio is not an easy thing. Some pioneer works, such as SpeechGPT [4], try to make a pre-trained LLAMA2 model generate speech. To realize this target, SpeechGPT expands its vocabulary with speech tokens and uses large-scale datasets and GPUs to learn the alignment between speech and text. In contrast, we propose to train an audio codec that can quantize the audio modality into LLM's token space, so that we do not need to expand the LLM's vocabulary like SpeechGPT.  **In summary, we aim to turn the powerful LLM into a universal audio understanding and generation model to tackle infinite audio tasks without training.** To realize this target, we propose to map the audio modality into LLM's token space. As a result, we only use 2 GPUs to train the codec model, and one GPU to conduct the inference with pre-trained LLMs.

**What is the advantage of this approach compared to using external TTS or speech enhancement modules?**

We thank this important comment and appreciate the reviewer. We agree that using external TTS or speech enhancement modules can also build a cascade system. However, our motivation is to build a universal end-to-end system, which supports audio and text as input and output. We expect our model can handle multiple tasks. We understand and agree with the reviewer's opinion: combining multiple external modules can also solve many tasks, such as HuggingGPT [6]. But we want to reach a consensus with the reviewer: combining multiple external modules or building a universal model (such as GPT4-o) are both potential paths toward artificial general intelligence (AGI). As a research work, we expect to explore more potential possibilities and inspire more work. 

**Furthermore, we also summarize all of the response Table and Figure as one PDF file**

[1] Wang C, Liao M, Huang Z, et al. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing[J]. arXiv preprint arXiv:2309.00916, 2023.

[2] Chu Y, Xu J, Zhou X, et al. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models[J]. arXiv preprint arXiv:2311.07919, 2023.

[3] Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//International conference on machine learning. PMLR, 2022: 12888-12900.

[4] Zhang D, Li S, Zhang X, et al. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities[J]. EMNLP, 2023.

[5] Borsos Z, Marinier R, Vincent D, et al. Audiolm: a language modeling approach to audio generation[J]. IEEE/ACM transactions on audio, speech, and language processing, 2023, 31: 2523-2533. 

[6] Shen Y, Song K, Tan X, et al. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face[J]. Advances in Neural Information Processing Systems, 2024, 36.

[7] Huang C, Lu K H, Wang S H, et al. Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 12136-12140.

[8] Ju Z, Wang Y, Shen K, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models[J]. ICML, 2024.

Meta-review: The paper presents LLM-Codec, an innovative audio codec model that utilizes Large Language Models (LLMs) to handle various audio tasks with minimal training data. By converting audio signals into the token space of LLMs, the model enables these language models to comprehend and generate audio content.

Four reviewers assessed this work, and there was a good discussion period. The authors have sorted out several key concerns raised by the independent reviewers, and some reviewers have modified their overall evaluation of the paper. Unfortunately, two key issues still remain unsolved, and both are acknowledge by the authors during the discussion phase, namely: (i) poor presentation -  it needs improvement, and (ii) current model's performance is worse than a common baseline, AudioLDM -  the authors' claim should be then tuned down. Nonetheless, all reviewers agree that the topic is interesting and the idea can be considered novel.
Decision: Accept (poster)