SALMONN: Towards Generic Hearing Abilities for Large Language Models
Authors: Changli Tang
Wenyi Yu
Guangzhi Sun
Xianzhao Chen
Tian Tan
Wei Li
Lu Lu
Zejun MA
Chao Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Multimodal large language models
speech and audio processing
music processing
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: SALMONN: Towards Generic Hearing Abilities for Large Language Models

Abstract: Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as 
automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning etc. SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning etc. The presence of cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. The source code, model checkpoints and data are available at https://github.com/bytedance/SALMONN.

Published as a conference paper at ICLR 2024
SALMONN: TOWARDS GENERIC HEARING ABILI- TIES FOR LARGE LANGUAGE MODELS
Changli Tang'; Wenyi Yu'; Guangzhi Sun', Xianzhao Chen', Tian Tan
Wei Li,, Lu Lu², Zejun Ma², Chao Zhanglt
Department of Electronic Engineering, Tsinghua University' ByteDance [tc120, ywy22 wttemmal.s.sssnsstee www.gnnunnu...... edu. cn, c22778tsinghua. www.ssiiinuua.edu....
ABSTRACT
Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general audi- tory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio in- puts and achieve competitive performances on number of speech and audio tasks used in training, such automatic speech recognition and translation, auditory- information-basee question answering, emotion recognition, speaker verification, and music and audio captioning etc. SALMONN also has diverse set of emer- gent abilities unseen in the training. which includes but is not limited to speech translation untrained languages, speech-based slot filling, spoken-query-bann question answering, audio based storytelling, and speech audio co- oo-reasoning etc. The presence of cross- mooal emergent abilities studied, and a novel few-shot ac- tivation tuning approach is proposed to activate such abilities. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards Al with generie hearing abilities. The source code, model checkpoints and data are available at ht https: /ttes://giiee //ghttub.bb//////// com/ bytpbacceeeae/eeeeeeeeeeeeaaaeee / SALMONN.
INTRODUCTION
Text-based large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023; Chiang et al., 2023; Anil et al., 2023; Du al., 2022) have demonstrated remarkable and even human-level perfor- mance in many natural language processing (NLP) tasks (OpenAI, 2023). Meanwhile, instruction tuning (Wei et al., 2022a; Chung et al., 2022: Ouyang et al., 2022: Peng et al., 2023), where data is organised pairs of user instruction (or prompt) and reference response, has emerged an LLM training paradigm that allows LLMs to follow open-ended user instructions. There a burgeoning research interest in empowering L.MS with multimodal perception abilities. Recent studies focus on connecting LLMS with either the encoder of one additional type of input, such as image (Li et al., 2023a; Alayrac al., 2022; Dai et al., 2023), silent video (Maaz et al., 2023; Chen al., 2023b; Zhao et al., 2022), audio events (Gong et al., 2023b; Lyu et al., 2023) or speech (Chen et al., 2023a), or the encoders of multiple input types together (Su al., 2023; Zhang al., 2023b). connection module and LLM adaptors can be used to align the encoder output spaces with the LLM input space, which are often trained by cross- cross-modal pre- graitrain and instruction tuning.
In this paper, we propose a speech audio language music open neural network (SALMONN). which a single audio-text multimodal L.MM that can perceive and understand three basic types of sounds including speech, audio events, and music. To enhance the performance on both speech and non- speech audio tasks, SALMONN uses a dual encoder structure consisting of a speech encoder from the Whisper speech model (Radford et al., 2023) and a BEAT's audio encoder (Chen et al., 2023c). A
"quual contribution "ooresponniig author

---

Published as a conference paper at ICLR 2024
window-level query Transformer (Q-Former) (Li et al., 2023a) is used the connection module to convert a variable-length encoder output sequence to a variable number of augmented audio tokens input to the Vicuna LLM (Chiang et al., 2023) and can achieve audio-text alignment with high tem- poral resolution. The low-rank adaptation (LoRA) approach (Hu et al., 2022) is applied to Vicuna as cross-modal adaptor align Vicuna's augmented input space with its output space and further improve its performance. A number of speech, audio, and music tasks are used in the cross-modal pre-training and instruction tuning stages of the window-level Q-Former and LoRA. The resulting multimodal LLMs can be confined to the specific types of tasks used in instruction tuning, par- ticularly speech recognition and audio captioning, and exhibit limited or no cross-moda emergent abilities, which we refer to as the lask over-fitling issue. In this paper, cross-modal emergent abilities refer to the abilities to perform cross-modal tasks unseen in training, which are essentially the emer- gent abilities of LLMS (Wei et al., 2022b) that are lost during instruction tuning. As a solution, we propose an extra few-shot activation tuning stage so that SALMONN regains the emergent abilities of LLMs and alleviates the considerable catastrophic forgetting to the trained tasks.
In order to evaluate SALMONN's cognitive hearing abilities, a wide range of speech, audio events, and music benchmarks are used. The tasks can be divided into three levels. The first level bench- marks eight tasks that are trained in instruction tuning, such as speech recognition, translation and audio captioning. while the other two levels benchmark untrained tasks. The second level includes five areech-basek NI.P tasks, such as translation to untrained languages and slot filling, which relies on multilingual and high-quality alignments between speech and text tokens. The last level of tasks, including audio-base based storytelling and speech audio co-reasoning etc, requires understanding not only speech but also non-speech auditory information. Experimental results show that SALMONN as a single model can perform all these tasks and achieve competitive performance on standard benchmarks, which reveals the feasibility of building artificial intelligence (AI) that can "hear" and understand general audio inputs consisting of mixtures of speech, audio events, and music.
The main contribution of this paper can be summarised as follows.
We propose SALMONN, the first multimodal L.M that can perceive and understand general audio inputs with speech, audio events, and music, to the best of our knowledge.
We study the presence of cross-modal emergent abilities by playing with the LoRA scaling factor, and propose a cheap activation tuning method as an extra training stage that can activate the cross- modal emergent abilities and alleviate catastrophic forgetting to tasks seen in training.
We evaluate SALMONN on range of tasks reflecting a degree of generic hearing abilities, and propose two novel tasks, audio-based storytelling and speech audio co-reasoning.
2 RELATED WORK
LLMs, as text-based dialogue models, have a fundamental connection with speech, and several studies have attempted to extend LLMs to support direct speech inputs with a connection module (Chen et al., 2023a; Wu et al., 2023; Fathullah et al., 2023; Yu et al., 2023; Huang et al., 2023a). To avoid the LLMS having overly long input speech token sequences caused by long-form speech inputs, different frame-rate reduction approaches have been developed, including backing-basee fixed-rate reduction approach (Fathullah et al., 2023; Yu et al., 2023), speech-recogniiioonbaaee based variable frame-rate reduction approach (Wu et al., 2023; Chen et al., 2023a), and Q-Former-basse approach with a fixed number of output frames (Yu et al., 2023) etc. When LLM-based speech synthesis is also considered, the LLM output space can be augmented with speech tokens as well, such as SpeechGPT (Zhang et al., 2023a) and AudioPaLM (Rubenstein et al., 2023).
Unlike speech, audio event inputs are often treated as fixed-sized spectrogram images that can be processed using visual-language LLM methods without explicitly modelling temporal correlations (Gong et al., 2023a;b; Zhang et al., 2023b). These methods are therefore unable to handle speech. Although Lyu et al. (2023) uses the speech encoder from the Whisper model, only audio event inputs are supported, which indicates the difficulty of the joint modelling of speech and audio events. With- out using LLMs, Narisetty et al. (2022) studies achieving speech recognition and audio captioning separately using the same model. Regarding music inputs, Liu et al. (2023) integrates the MERT music encoder (Li et al., 2023b) with an LLM for music understanding tasks. AudioGPT allows text-based LLM to process speech, audio events, and music by interacting with other models in
2

---

Published as a conference paper at ICLR 2024
pipeline based set of pre-defined tasks (Huang al., 2023b). Compared with AudioGPT, SALMONN is an end-to-end model with cross-modal emergent abilities for open-ended tasks.
In addition audio, multimodal LLMs are more widely studied in visual modalities, such image (Zhu et al., 2023; Li et al., 2023a), video (Maaz et al., 2023; Chen et al., 2023b) and audio-visual (Su et al., 2023; Lyu et al., 2023; Sun et al., 2023). Modality alignment in those models is often achieved via either a fully connected layer or an attention- -based module. In particular, the Q-Former structure (Li al., 2023a) used by SALMONN is commonly applied visual modalities, such as in MiniGPT-4 (Zhu et al., 2023), InstructBLIP (Dai et al., 2023), Video-LLaMA (Zhang et al., 2023b).
3 METHODOLOGY
The model architecture of SALMONN is introduced in Section 3.1. Our training method is presented in Section 3.2, which includes the pre-training and fine-tuning stages, and the proposed activation tuning stage solution to the task over-fitting issue.
3.1 MODEL ARCHITECTURE
The model architecture of SALMONN is shown in Fig. 1. The output features of the two comple- mentary auditory encoders are synchronised and combined. Q-Former is used as the connection module and applied at the frame level, whose output sequence is integrated with the text instruction prompt and fed into the LLM with LoRA adaptors to generate the text response.
Dual Auditory Encoders: A speech encoder from OpenAl's Whisper model (Radford et al., 2023) and a non-speech BEAT' audio encoder (Chen et al., 2023c) are used. The Whisper model is trained for speech recognition and translation based on large amount of weakly supervised data, whose encoder output features are suitable to model speech and include information about the background noises (Gong et al., 2023a). BEATs is trained to extract high-level non-speech audio semantics information using iterative self-supervised learning. The input audio is first tokenised then masked and predicted in training. The tokeniser is updated by distilling the semantic knowledge of the audio tokens (Chen et al., 2023c). Therefore, the resulting auditory features of these two encoders are complementary and suitable for general audio inputs with both speech and non-speech information.
Since both encoders have the same output frame rate of 50Hz, the concatenated output features are
Z Concat (Encodee X), Encoderheats (X))
(1)
where X is variable-length length general audio input sequence, Encoderwhisper() and Encoder (-) are the Whisper and BEATs encoder, Concat(-) is the frame-by-frame concatenation operation along the feature dimension, Z is the concatenated encoder output sequence with T frames.
The Q-Former structure is commonly used to convert the output of an image encoder into a fixed number of textual input tokens of an LLM (Li et al., 2023a), which re- quires modification when applied to handle audio inputs of variable lengths. Specifically, regarding the encoder output of an input image as Z, Q-Former employs fixed number of N trainable queries Q to transform Z into N textual tokens H; using a number of stacked Q-Former blocks. A Q-Former block is similar to Transformer decoder block (Vaswani et al., 2017), apart from the use of a fixed number of trainable static queries Q in the first block and the removal of the causal masks from the self-attention layers. In this way, Q-Former allows the queries in Q to refer to each other first using a self-attention layer and then interact with Z, using cross-attention layers.
Window-level Q-Former:
Regarding variable-length general audio input with Z [2,) by segmenting Z into L-sized windows and padding the last window with zeros, it becomes [[Z, 1 | (=(111)x [T/L]
instead of using Q-Former at the sequence level to convert the entire Z into N textual tokens. SALMONN uses Q-Former at the window level if the encoder output frames stacked in each window were an image. As result, the textual token sequence H becomes
[TyL] = [Q-Former(( Q, [T/L]
(2)
where Q-Fommer()) is the Q-Former function and H has [T/L] x N textual tokens. The window- level Q-Former uses variable number of textual tokens and is more efficient for variable-length
3

---

Published as a conference paper at ICLR 2024
Text response
Q-Former queries
Whisper features
BEATs features
Large Language Model
LoRA
Auditory embeddings
Textual embeddings
Window-level
Q-Foreer
Text instruction prompt
Whisper
Encoder
BEATs
Encoder
Figure 1:
The model architecture of SALMONN. A window-level Q-Former is used as the nection module to fuse the outputs from a Whisper speech encoder and a BEATs audio encoder as augmented audio tokens, which aligned with the LLM input space. The LoRA adaptor aligns the augmented LLM input space with its output space. The text prompt is used to instruct SALMONN answer open-ended questions about the general audio inputs and the answers are in the LLM text responses. The LLM and encoders are kept frozen while the rest can be updated in training.
sequences. In addition, H is enforced have monotonic alignment with Z, resulting in better temporal resolution which is important for speech recognition.
LLM and LoRA: A pre-trained Vicuna LLM is used this work (Chiang al., 2023) which is a LLaMA LLM (Touvron et al., 2023) fine-tuned to follow instructions. LoRA (Hu et al., 2022) widely used pfrfieter-efficient fine-tuning method for LLM adaptation, which is used in SALMONN to adapt the query and value weight matrices in the self-attention layers of Vicuna. In this work, LoRA trainable while Vicuna not.
3.2 TRAINING METHOD
three-stage cross-modal training method of SALMONN introduced in this section. Besides the pre-training and instruction tuning stages used by recent visual LLMs (Dai et al., 2023; Zhang al., 2023b), an additional activation tuning stage is proposed to resolve the issue of over-fitting to the speech recognition and audio captioning tasks in instruction tuning.
To mitigate the gap between the pre-trained parameters (LLM and encoders) and the randomly initialised parameters (connection module and adaptor), a large amount of speech recognition and audio captioning data is used to pre-train the window-level Q-Former and LoRA. These two tasks contain key auditory information about the contents of speech and non-speech audio events, and both do not require complex reasoning and understanding and therefore can help SALMONN learn high-quality alignment between the auditory and textual information.
Pre-training Stage:
Instruction Tuning Stage: Similar to NLP (Wei et al., 2022a) and visual-language (Dai et al., 2023), audio-text instruction tuning with list of supervised speech, audio event, and music tasks, as shown in Section 4.2 and Table 1, is used the second stage of SALMONN training. The tasks are selected based on their importance (e.g. speech recognition and audio captioning) and the indispensability of having such ability in tests (e.g. overlapping speech recognition, phone recognition and music captioning). The instruction prompts are generated based on the texts paired with the audio data.
Task Over-fitting: Although SALMONN built with only the first two stages of training can produce competitive results on the tasks trained in instruction tuning, exhibits limited or almost no ability to perform untrained cross modal tasks, especially the tasks that require cross-moke modsl co-reasoning abilities. In particular, the model sometimes violates the instruction prompts and generates irrelevant responses if it received instruction related to task commonly seen in training (e.g. speech
4

---

Published as a conference paper at ICLR 2024
recognition). We refer to this phenomenon task over-fitring A theoretical analysis of the issue is presented here and detailed experimental verification is provided in Sections 5.3 and 5.4.
We attribute task over-fitting to two reasons. First, compared to the text-only data used in LLM training. only simpler instruction prompts are used in our cross-modal instruction tuning (Wei et al., 2022a) and the resulting responses not complex and diverse. Meanwhile, some tasks included in instruction tuning, in particular speech recognition and audio captioning, have more determinis- tic outputs than the other tasks, such as speech and audio question answering. These two reasons combined to cause the intrinsic conditional language model (LM) to bias to an ill-formed distribu- tion with poor generalisation ability, which prohibits SALMONN from performing untrained cross- modal tasks. More specifically, at test time the response text sequence Y of a test input X given a new instruction prompt I can be generated according to Y = arg maxy PA PX I), which is also the objective to maximise in training. Using the Bayes' Rule, there is
IX = PA((X)) -
(3)
Since only limited text responses are seen in SALMONN training. the intrinsic conditional LM Pa(Y is biased towards the Y sequences that are strongly-aligned with X, such as the tran- scriptions of automatic speech recognition (ASR) and automatic audio captioning (AAC) tasks, especially simple and short transcriptions. From Eqn. (3), this causes I', zero-shot instructions with more diverse responses, to have small PA PY
Activation Tuning Stage: An effective approach to alleviating task over-fitting is to regularise the intrinsic conditional LM PA (Y An easy way to achieve this is to fine-tune SALMONN on tasks with longer and more diverse responses, such as auditory-ifformation- storytelling. Paired training data for such tasks can be generated based on the text paired with speech recognition or audio and music caption data, either manually by human annotators or automatically by prompting
question answering and
LLM.
We use an efficient approach that can enable SALMONN to generate long and diverse responses for zero-shot instructions by simply reducing the scaling factor of the LoRA adaptor. This is an alter- native way to regularise PA (Y(X) since the intrinsic conditional LM can only be learned with the window-level Q-Foreer and LoRA, since they are the only modules that are updated in training. The effect of reducing the LoRA scaling factor can be found in Section 5.2, which can indeed activate question-answering and storytelling abilities and produce long and diversified responses but also considerably degrade the results on the trained tasks. To retain competitive results while restoring the cross modal emergent abilities, a discounted LoRA scaling factor to perform the third stage of fine-tuning termed activation tuning. Experimental results showed later in Section 5.4 demonstrate that activation tuning is efficient and effective few-shot self-supervised training approach.
propose to the responses generated by SALMONN with
4 EXPERIMENTAL SETUP
4.1 MODEL SPECIFICATIONS
SALMONN uses the encoder part of Whisper-Laree-22 (Radford et al., 2023) model as the speech encoder. the fine-tuned BEATs (Chen et al., 2023c) encoder as the audio encoder, and Vicuna LLM with 13 billion parameters (Chiang et al., 2023) as the backbone LLM. For the window- level Q-Former, approximately 0.33 seconds per window. This leads to 88 textual tokens output by Q-Former for a 30-second audio. Regarding the hyper-parameters of LoRA (Hu al., 2022), we set the rank 8 and the scaling factor to 4.0. Only the parameters of Q-Former and LoRA are updated in training, which resulted in 33 million (M) parameters that is ~0.24% of the entire SALMONN model.
N 1 resulting in only one trainable query, and use L 17 which is
4.2 DATA SPECIFICATIONS
The three-stage training proposed in Section 3.2 is used. The data used for the first pre-training stage consists of both 960-hour LibriSpeech training set (Panayotov et al., 2015) and 1000-hour GigaSpeech M-set (Chen et al., 2021) for speech recognition, well 2800-hour WavCaps (Mei et al., 2023) (with audio clips longer than 180 seconds removed), AudioCaps (Kim et al., 2019) and Clotho (Drossos al., 2020) datasets for audio captioning.
5

---

Published as a conference paper at ICLR 2024
The second instruction tuning stage involves multiple tasks, including ASR, automatic speech trans- lation (AST), AAC, phone recognition (PR), emotion recognition (ER), music captioning (MC), overlapped speech recognition (OSR), speaker verification (SV), gender recognition (GR) and speech question answering (SQA), audio question answering (AQA) and music question answering (MQA). In the SQA, AQA and MQA tasks, the questions are generated based the text caption labels using ChatGPT, and the model needs to provide answers based on the general audio input and the text prompt with a question. The data used in this stage is listed in Table 1, where "En2Zh" refers to AST from English to Chinese
For the final activation tuning stage, twelve stories were written based on the audio clips by SALMONN with a reduced LoRA. Then the model is trained using seacher-forcing-ae cross- entropy training for 12 steps, with each step using only one story sample, when activating the cross- modal emergent abilities of SALMONN.
Table I: Training data used in the cross-modal instruction tuning stage.
Task
Data Source
#Hours
#Samples
ASR
LibriSpeech + GigaSpeech
960 + 220
280K + 200K
En2Zh
CoVoST2-En27th (Wang et al., 2021)
430
290K
AAC
AudioCaps + Clotho
130 24
48K + 4K
PR
LibriSpeech
960
280K
ER
IEMOCAP Session 1-4 (Busso et al., 2008)
5
4K
MC
MusicCaps (Agostinellii et al., 2023)
14
3K
OSR
LibriMix (Cosentino et al., 2020)
260
64K
SV
VoxCeleb1 (Nagrani et al., 2019)
1200
520K
GR
LibriSpeech
100
28K
SQA
LibriSpeech
960
280K
AQA
WavCaps + AudioCaps
760 + 130
270K + 48K
MQA
MillionSong' + MusicNet (Thickstun et al., 2017)
400 +
48K 0.3K
Total
~4400
2.3M
4.3
TASK SPECIFICATIONS
Since text LLMs have the abilities of zero-shot learning via instruction tuning (Wei et al., 2022a), the emergence of such abilities is expected when high-quality cross-modal alignment is achieved when connecting the backbone text-based LLM with multimodal encoders. To evaluate the zero- shot cross-modal emergent abilities of SALMONN, 15 types of speech, audio, and music tasks are selected and divided into three different levels.
Task level consists of the tasks used in instruction tuning and are therefore easiest for SALMONN to perform. The list of such tasks and their training data are given in Section 4.2, and the evaluation metrics for each task are presented in Table 2.
Task level 2 includes untrained tasks and is therefore more difficult than level 1. The level 2 tasks speech-based NLP tasks including speech keyword extracting (KE), which evaluates the accuracy of the keywords extracted based on the speech content; spoken-query-bawee (SQQA), which evaluates the common sense knowledge retrieved based on the question in speech; speech-based slot filling (SF) that evaluates the accuracy of the required slot values, usually named entities, obtained from the speech content. Two AST tasks, En2De (English to German) and En2Ja (English to Japanese) are also included, which are also considered as cross-modal emergent abilities since only En2Zh is trained in instruction tuning, Vicuna, the backbone LLM of SALMONN, can perform all level 2 tasks based on speech transcriptions. Therefore, SALMONN to achieve such tasks based speech in fully end-to-end way without requiring any explicit speech recognition.
question answering
Task level has the most difficult tasks including audio-bases storytelling (Story) and speech audio co-reasoning (SAC). basio storytelling is write a meaningful story based the auditory information from the general audio inputs. SAC requires the model to understand a spoken question embedded in the input audio clip, find evidence from the background audio events music, and

---

Published as a conference paper at ICLR 2024
reason from it to answer the question. Both level 3 tasks are new tasks that are first proposed in this paper, to the best of our knowledge, which requires SALMONN to perceive speech, audio, and music, and understand and reason based on the auditory information in fully end-to-end way.
Table lists all test sets and their evaluation metrics. Following rate (FR) is an extra metric used successfully for some level 2 and level 3 tasks, which measures the percentage that SALMONN follow the instructions. FR is considered since the selected tasks are complex and easier to suffer from the violation of instructions caused by task over-fitting. It worth noting that we only calculate the diversity metric of the Story task by counting the number of different words in the story, which simply represents the richness of the story instead of the quality.
Test sets, metrics, and sources of the reference values used in the three levels of tasks. The speech data used in SQQA and KE are synthesised using commercial text-to-speech product. For reference values: "Whisper" refers to using Whisper-L.rge- for ASR, "Whisper + Vicuna" refers to feeding the Whisper-Large-22 ASR output transcriptions into Vieuna, and the rest the state-of-the-art results to the best of our knowledge.
Table 2:
Task
Test Data
Eval Metries
Reference Value
ASR
LibriSpeech fest-clean/-ohher,
%WER
Whisper
ASR
GigaSpeechh test CoVoST2-En224
&WER
Whisper
En2Zh
BLEU4
(Wang et al., 2021)
AAC
AudioCaps
METEOR SPIDEr
(Mei et al., 2023)
PR
LibriSpeech test-clean
%PER
WavLM (Chen et al., 2022)
ER
IEMOCAP Session
Accuracy
(Wu et al., 2021)
MC
MusicCaps
BLEU4, RougeL
(Doh et al., 2023)
OSR
LibriMix
%WER
(Huang et al., 2023c)
SV
Voxceleb1
Accuracy
En2De
CoVoST2-En2De
BLEU4
Whisper Vicuna
En2Ja
CoVoST2-En2Ja
BLEU4
Whisper Vicuna
KE
Inspec (Hulth, 2003)
Accuracy
Whisper Vicuna
SQQA
WikiQA (Yang et al., 2015)
Accuracy (FR)
Whisper Vicuna
SF
SLURP (Bastianelli et al., 2020)
Accuracy (FR)
Whisper + Vicuna
Story
AudioCaps
Diversity (FR)
SAC
In-house Data
Accuracy (FR)
5
EXPERIMENTAL RESULTS
5.1
FULL RESULTS ON ALL 15 TASKS
The results on all 15 tasks produced by SALMONN are shown Table 3. From the results, SALMONN, without or with activation tuning, can produce competitive results on all level tasks. However, the model without activation tuning suffers severely from task over-fitting and can barely perform level 2 and level tasks. In particular, in SQQA, Story, and SAC, where multimodal interac- tions are emphasised. SALMONN without activation tuning can hardly follow the instructions. The FRs of performing SQQA. SE, Story and SAC tasks improve considerably with activation tuning.
Fig. 2 illustrates the trends of model performance change on ASR & PR, SQQA, Story, and SAC different training steps of activation tuning, which reveals that activation tuning only needs a few training samples and steps to activate the emergent abilities: the results of ASR and PR remain almost unchanged, while the results of SQQA, Story and SAC have an emergent trend.
More detailed analysis about the results are discussed in Appendix B due to page limit.
5.2 DISCOUNTING LORA SCALING FACTOR
This section explores the influence of the use of timt-time discounting of the LoRA scaling factor for alleviating the task over- fittiniting issue without activation tuning. As shown in Fig. 3, when the LoRA scaling factor decreases to around 2.0, i.e. to half of its original value, the model suddenly emerges

---

Published as a conference paper at ICLR 2024
Table 3: Results of all 15 tasks produced by SALMONN without & with activation tuning (w/o & w/ Activation). The ASR results are presented in a tuple with three the WWER evaluated on 3 test sets, namely (LibriSpeech test-clean, LibriSpeech lest-other, GigaSpeech).
Method
ASR
En2Zht
AACT
PR
ERT
MC1
OSR
SVT
w/o Activation
(2.1, 4.9, 9.1)
34.4
25.6
47.6
4.2
0.63
3.5, 22.1
20.7
0.93
w/ Activation
(2.1, 10.0)
33.1
24.0
40.3
4.2
0.69
5.5, 21.8
23.0
0.94
Reference Value
(2.2, 5.1, 9.2)
38.9
25.0
48.5
3.1
0.81
21.5
7.6
(a) Results of the level tasks.
Method
En2Det
En2Jat
KE
SQQA
SF1
Storyt
SAC1
w/o Activation
19.7
22.0
0.30
0.19 (0.29) 0.33 (0.77)
7.77 (0.00)
0.02 (0.04)
w/ Activation
18.6
22.7
0.32
0.41 (0.98) 0.41 (0.99)
82.57 (1.00)
0.50 (0.73)
Reference Value
16.5
15.6
0.31
0.77 (1.00) 0.46 (1.00)
(b) Results of the level 2 and level 3 tasks.
3.0
1.00
1.00
2.5
4,5
0.75
0.75
75
0.75
0.75
0.75
2.0
0.50
0.50 g
50-
0.50 E
0.50
0.50
1.5
3.5
0.25
0.25
0.25
0.25
S00A
Story
1.0
2
10 12
10 12 Number of Stops
0.00
12
Number of Stops
Number Stops
Number of Stops
(a)
(b)
(c)
(d)
Figure 2: Performance changes on ASR & PR (a), SQQA (b), Story (c) and SAC (d) along with the FR of the emergent abilities against the number of training steps during activation tuning.
with modal reasoning abilities, which, together with the drops in %PER, proves the existence of the intrinsic conditional LM embedded in LoRA.
100
1.00
1.00
40
75
0.75
0.75
75-
0.75
0.75
0.75
SWER
0.50
0.50
50
0.50 g
0.50
0.50
20
0.25
0.25
ASA
500A RK4
Stry Ds
3
3
2
1
LoRA Scaling
LoRA Scaling
LoRA Scaling
LoRA Scaling
(a)
(b)
(c)
(d)
Figure 3: Performance changes on ASR & PR (a), SQQA (b), Story (c) and SAC (d) together with
the FR of the emergent abilities when discounting the LoRA scaling factor test time.
5.3 ANALYSIS OF TASK OVER-FITTING AND ACTIVATION TUNING
To verify the influences of the intrinsic conditional LM, calculate the perplexities (PPLs) of PA(Y I) and (Y) of each step of the activation tuning stage. In detail, for given audio X, the probability of the Y sequence corresponding to the probed task is calculated using teacher forcing based on the text instruction prompt of a certain task or without using any I.
As shown Fig. 4. PPLs were compared between the ground -tuth responses of the task of Story/SAC (generated by discounting the LoRA scaling factor) and the responses of the task that the model performed incorrectly due task over-fitting (AAC in this example). In sub-figures (a) and (b), no text instruction prompt was used when computing PA PA(Y (Y)X), showing that without ac- tivation tuning the PPL the Y of AAC was obviously lower than that the of Story/SAC. During activation tuning, the PPL gaps between these tasks were mitigated, meaning the bias to the dominant AAC task was considerably reduced. In sub-figures (c) and (d), we probed PA (Y I), the PPLs with the instructions of Story and SAC respectively. It is revealed that before activation tuning, the PPL of AAC is lower than that of Story/SAC even if the text instruction prompt is to per- form Story/SAC, which explains the failure of the model to follow the instruction. During activation
8

---

Published as a conference paper at ICLR 2024
tuning, the PPL values of the Y of Story/SAC gradually become lower than those of AAC, and the model can eventually perform the instructed task.
4.0
2.50
AAP mesponse-Story
Aes respensieSAC
Asppense-AAC ree repornsenStary
PesparseeSAC.
g
2:25
8
3.0
2.00
2.5
1.75
2.0
6
12
12
8 10 12
10 12
Number Steps
Number of Stops
Number of Steps
Number Stops
(a)
(b)
(c)
(d)
X) for AAC and Story, (b) shows - In ( for AAC and SAC, (c) shows - In ( Story), and (d) shows In PA Y ( SAC).
Figure 4: Changes in PPL during the activation tuning stage. (a) shows - In PA
5.4 ACTIVATION TUNING USING DIFFERENT TASKS AND DATA
We have explored several activation methods to activate the model, including training on long stories written based on the audio, text-based question question-answeriing answering (QA) task pairs with long answers, and the ASR task with long speech transcriptions. Both LoRA and Q-Former are fine-tuned with these three methods. By ignoring the Q-Former and audio inputs, Vicuna and LoRA are fine-tuned as an adapted text-based LLM. As another control group, the experiment of using both LoRA and Q-Former but without tuning the Q-Former is also conducted.
The results are shown in Table 4. From the results, neither ASR (Long), training ASR with long labels, nor Story (Text based), training LoRA based on only text prompt input, can activate the model. Training with stories QA with long answers, Story or QA (Long), fact that the ASR task emphasises improving high-quality cross- cross- modal alignments that make the distribution even more biased towards the intrinsic conditional LMM As for text-based fine-tuning on stories, this actually affect P I) I) instead of P(Y[X, I). where T, is the reference text of the speech. Therefore, text-based training cannot alleviate task over-fitting. QA data with long answers can activate the LLM, but the performance of the activated model is worse than that activated using the Story task, which especially results in a higher repeat rate of the response generation. That possibly due to the fact that the paired answer to the QA task is less diverse than the Story task, which leads to less effective regularisation of the intrinsic conditional LM PA XY
This lies in the
Table 4: Results of selected tasks with activation tuning performed based on different tasks. "Repeat Rate" is the percentage of test samples that SALMONN generated responses repeatedly.
Activation Method
ASR
PR
SQQA
Storyt
SACT
Repeat Ratel
w/o Activation
2.1
4.2
0.19 (0.29)
7.77 (0.00) 0.02 (0.04)
0.2%
Story
2.1
4.2
0.41 (0.98)
82.57 (1.00)
0.50 (0.73)
0.1%
QA (Long)
2.1
4.3
0.40 (0.93)
59.82 (1.00)
0.34 (0.71)
4.6%
ASR (Long)
2.2
4.2
0.22 (0.28)
7.87 (0.00)
0.12 (0.03)
0.1%
Story (Text based)
2.1
4.2
0.23 (0.32)
8.45 (0.03)
0.11 (0.03)
0.1%
Story (LoRA only)
2.1
4.2
0.44 (0.96)
82.29 (1.00)
0.34 (0.65)
0.2%
6 CONCLUSION
This work proposes SALMONN, speech audio language music open neural network that can be re- garded as a step towards generic hearing abilities for LLMs. Equipped with dual auditory encoders, SALMONN achieved competitive performances
trained tasks including speech recognition, dio captioning and speech translation etc., while generalising to a range of untrained understanding tasks such slot filling, speech translation for untrained languages and keyword extracting. More- over, a proposed activation tuning stage enables SALMONN with remarkable emergent abilities, such as audio-based storytelling and speech audio co-reasoning. As a result, with thorough and comprehensive experimental evaluations, SALMONN has demonstrated a promising direction for developing generic hearing Al in the future.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
To make the experiments and models reproducible, the training data and the benchmark details are provided in Section 4.2 and Section 4.3. The source code, model checkpoints, and data are released on the SALMONN project GitHub page.
REFERENCES
Andrea Agostinelli. Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv:2301. 1325, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, al. Flamingo: visual language model for few-shot learning. In Proc. NewrIPS, New Orleans, 2022.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. PaLM technical report. arXiv:2305. 10403, 2023.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A spoken language understanding resource package. In Proc. EMNLP, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proc. NeurlPS, New Orleans, 2020.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-LLM: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv:2305.041 04160, 2023a.
Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Qiao, Tong Lu, et al. VideoLLM: Modeling video sequence with large language models. 1322: 2023b.
Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio. In Proc. Interspeech, Brno, 2021.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka. Xiong Xiao, et al. WayLM: Large-scale self- self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6): 1518, 2022.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Proc. ICML, Honolulu, 2023c.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. Utthttps:
//lmsys....///////// wrt./orgg/boo..ooo.......... httg://log///00 03 30 1:000-iie viiiviana/.....
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, et al. Scaling instruction- finetuned language models. arXiv:2210.1 14116,,1114,,, 2022.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. LibriMix: An open-source dataset for generalizable speech separation. arXiv:2005. /262, 2020.
10

---

Published as a conference paper at ICLR 2024
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, et al. InstructBLIP: Towards general- purpose vision- saionnlanguage models with instruction tuning. (rXiv:2305.06500, 2023.
SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. LP-MusicCaps: LLM-based pseudo music captioning. arXiv:2307. 16372, 2023.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In Proc. ICASSP, Barcelona, 2020.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proc. ACL, Dublin, Ireland, 2022.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. arXiv preprint arXiv:2307. 1795, 2023.
Yuan Gong, Sameer Khurama, Leonid Karlinsky, and James Glass. Whisper-AN AT: Noise-robust au- tomatic speech recognizers are also strong general audio event taggers. In Proc. Interspeech, Dublin, Ireland, 2023a.
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv:2305. 10790, 2023b.
Haohan Guo, Shaofei Zhang, Frank K. Soong, Lei He, and Lei Xie. Conversational end-to-end TTS for voice agents. In Proc. SLT, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of large language models. In Proc. ICLR, 2022.
Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi- Yuan Hsiao. Chun-Yi Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-SPPRRB: To- wards dynamic, collaborative, and comprehensive instruction-tuniin benchmark for speech. arXiv:2309.09510 2023a.
Rongjie Huang. Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang. Jinglin Liu, al. AudioGPT: Understanding and generating speech, music, sound, and talking head. arXiv:2304.1 /2995, 2023b.
Zili Huang, Desh Raj, Paola García, and Sanjeev Khudanpur. Adapting self-supervised models to multi-talker speech recogmition using speaker embeddings. In Proc. ICASSP. Rhodes, Greek, 2023c.
Anette Hulth. Improved automatic keyword extraction given more linguistic knowledge. In Proc. EMNLP, Sapporo, Japan, 2003.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Proc. NAACL-HLT, Minneapolis, 2019.
Harlen Lane and Bernard Tranel. The Lombard sign and the role of hearing in speech. Journal of Speech Hearing Research, 14:677-709, 1971.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. ICML, Hawaii, 2023a.
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et MERT: Acoustic music understanding model with large-scale self-supervised training. arXiv:2306.0001 2023b.
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding LLaMA: Advancing text-to-music generation with question answering and captioning. arXiv:2308. 11276, 2023.
11

---

Published as a conference paper at ICLR 2024
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang. Bingshuai Liu, et al. Macaw- LLM: Multi-modal language modeling with image, audio, video, and text integration. arXiv:2306.0909) 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChaIGPT: Towards detailed video understanding via large vision and language models. arXiv:2306.05424, 2023.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumb- ley, Yuexian Zou, and Wenwu Wang. WavCaps: A CharGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv:2303.13395 2023.
Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. Voxceleb: Large-scale speaker verification in the wild. Computer Speech Language, 60:101027, 2019.
Chaitanya Narisetty, Emiru Tsunoo, Xuankai Chang, Yosuke Kashiwagi, Michael Hentschel, and Shinji Watanabe. Joint speech recognition and audio captioning. In Proc. ICASSP, Singapore, 2022.
OpenAl. GPT-4 technical report. arXiv:2303.08774,222 2023.
Pilar Oplustil-Gallegos, Johannah O' O'Mahony, and Simon King. Comparing acoustic and textual representations of previous linguistic context for improving text-to-speech. Proc. SSW, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Proc. NeurlPS, New Orleans, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proc. ICASSP, South Brisbane, 2015.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv:2304.03277, 2023.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. Proc. ICML, Honolulu, 2023.
Paul K. Rubenstein, Chulayuth Asawaroengchai, Due Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. AudioPaLM: A large language model that can speak and listen. 1xXv:23306.1225. 2023.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One model to instruction-follow them all. urXiv:2305. 16355, 2023.
Guangzii Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Fine-grained audio-visual joint representations for multimodal large language models. arXiv preprint arXiv:2310.055 2023.
John Thickstun, Zaid Harchaoui, and Sham Kakade. Learning features of music from scratch. In Proc. ICLR, Toulon, France, 2017.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, et al. LLaMA: Open and efficient foundation language models. arXiv:2302. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, Long Beach, 2017.
Changhan Wang, Anne Wu, and Juan Pino. CoVoST and massively multilingual speech translation. In Proc. Interspeech, Brno, Czech Republic, 2021.
Jason Wei, Maarten Bosma, Vincent Y Zhao. Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoc Le. Finetuned language models are zero-shot learners. In Proc. ICLR, 2022a.
12

---

Published as a conference paper at ICLR 2024
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022b.
Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. arXiv:2307.039917 2023.
Wen Wu, Chao Zhang, and Philip C Woodland. Emotion recognition by fusing time synchronous and time asynchronous representations. In Proc. ICASSP, Toronto, Canada, 2021.
Guanghui Xu, Wei Song, Zhengchen Zhang, Chao ZHang, Xiaodong He, and Bowen Zhou. Improv- ing prosody modelling with cross-utterance BERT embeddings for end-to-end speech synthesis. In Proc. ICASSP, 2021.
Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Proc. EMNLP, Lisbon, Portugal, 2015.
Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for asr. arXiv preprint Sttv:2309.139633 2023.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abili- ties. / 1000, 2023a.
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-turne audio-visual language model for video understanding. arXiv:2306.0288 028588 2023b.
Ya-Jie Zhang, Chao Zhang, Wei Song, Zhengchen Zhang, Yonghui Wu, and Xiaodong He. Prosody modelling with pre-trained cross-utterance representations for improved speech synthe- sis. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 31:2812 2823,,,, 2023c.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proc. CVPR, New Orleans, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2024
A
EXAMPLES OF SALMONN
SALMONN is capable of a large range of auditory tasks. Here we list some examples in Fig. 5-21.
(A man asks "Can you guess where am right now", and explosions and gunshots can be heard in the background.)
Recognize the speech and give me the transcription.
USER
Can guess where I am right now?
SALMONN
Figure 5: Automatic Speech Recognition
(A man asks "Can you guess where am right now", and explosions and gunshots can be heard in the background.)
Listen to the speech and translate it into German.
USER
Kannst du erraten, wohin ich gerade fahre?
SALMONN
Figure 6: Automatic Speech Translation (En2De)
(A man asks "Can you guess where am right now", and explosions and gunshots can be heard in the background.)
Provide the phonetic transcription for the speech.
USER
sil K AHO Y UWI G EHI S W EHI RAYI EYI EHI MR AWI sil
SALMONN
Figure 7: Phone Recognition
14

---

Published as a conference paper at ICLR 2024
(A man asks "Can you guess where I am right now", and explosions and gunshols can be heard in the background.) )
Please describe the audio.
USER
person speaking while gunshots are being fired in the background.
SALMONN
Figure 8: Automatic Audio Captioning
(A man asks "Can you guess where am right now", and explosions and gunshots can be heard in the background.) )
Recognize what the speaker says and describe the background audio at the same time.
USER
The speaker says "Can you guess where I am right now?" while the background audio has gunshots and explosions.
SALMONN
Figure Joint Speech and Audio Captioning
(A man asks "Can you guess where right now", and explosions and gunshots can be heard in the background,
Please answer the speaker's question in detail based on the background sound.
USER
Based on the background sound, it seems like the speaker is in war zone or a combat situation. The sound of gunfire and explosions can be heard in the background. The speaker is asking if the listener guess where they arc.
SALMONN
Figure 10: Speech Audio Coreasoning
15

---

Published as a conference paper at ICLR 2024
(Birds chirping, a woman singing, man speaking and duck quacking.)
Please list each event in the audio in order.
USER
1. Birds chirping in the distancc.
2. woman singing.
man speaking.
4. duck quacking.
SALMONN
Figure 11: Audio Event Detection
(Birds chirping, woman singing, speaking and duck quacking.)
Based the audio, write story in detail. Your story should be highly related the audio.
USER
It was bcautiful day in the countryside. The sun was shining, the birds were singing, and the wind was blowing gently through the fields. In the midst all this natural beauty, group of ducks were swimming in pond.
they swam, one of the ducks began to quack in unique way. was as if the duck was trying to say something important. The other ducks noticed this and began to listen.
Suddenly, appeared on the edge of the pond. He had heard the duck's quacking and had come to see what was going on. The duck continued to quack, and the man began to speak...
SALMONN
Figure 12: Audio Story Telling
(Birds chirping, a woman singing, man speaking and duck quacking.)
How many speakers did you hear this audio? Who are they?
USER
heard two speakers in this audio. One is man and the other is woman.
SALMONN
Figure 13: Speaker Recognition
16

---

Published as a conference paper at ICLR 2024
(A man says excitedly: "All right, let's do".)
Describe the emotion of the speaker.
USER
The speaker is excited.
SALMONN
Figure 14: Emotion Recognition
(A person asks "What is the highest mountain in the world".)
Please answer the question in detail.
USER
The highest mountain in the world is Mount Everest, located in the Himalayas between Nepal and Tibet. It stands at height of 8,848 meters (29,029 feet) above level.
SALMONN
Figure 15: Spoken-query-ie based Question Answering
(A part of Steve Jobs' speech: "Your time limited so don't waste it living someone else's life. Don't be trapped by dogma which is living with the results of other people's s thinking. Don't the noise of other's opinions drown out your follow your heart and intuition. They somehow already know what you truly want to become everything else secondary. When was young there was an amazing publication called the whole earth catalog, which one of the bibles of my"")
di
inner voice. And most important have the courage to
Give only three keywords of the text. Explain your reason.
USER
The keywords are 'time', 'limited', and 'courage'.
The reason for choosing these keywords is that the speaker emphasizes the importance of not wasting time living someone else's life, not being trapped by dogma, and having the courage to follow onc's own inner voice.
SALMONN
Figure 16: Keywords Extracting
17

---

Published as a conference paper at ICLR 2024
(A girl says: "Schedule meeting with the accounting department on half past two on Friday")
What is the time mentioned in the speech?
USER
Half past Friday
SALMONN
Figure 17: Slot Filling
(It's soothing, elegant piano piece. At the end of the audio woman begins to sing.)
Please describe the music detail.
USER
This is beautiful piano and vocal piece. The piano plays simple melody while the female vocalist sings in high register. The atmosphere is romantic and dreamy. The piece has soft and mellow feel This piece could be used in romantic movie scene or lullaby.
SALMONN
Figure 18: Music Captioning
(It's a soothing, elegant piano picce. At the end of audio woman begins sing.)
What is the emotion of the music? Explain the reason in detail.
USER
The music emotional and evokes feelings love affection. The reason for this is that the music a love song, with lyrics that express the deep emotions and feelings of love and affection. The music also instrumental, which means that it relies solely on the melody and harmony emotions. This creates more intimate and personal experience for the listener, allowing them to interpret the music in their own way and connect with it on a deeper level.
instruments to convey these
SALMONN
Figure 19: Music Emotion Recognition
18

---

You't re the one I've been searching for, Published as a conference paper at ICLR 2024
(It's soothing, elegant piano piece. At the end of the audio woman begins to sing.)
Can you write some lyrics of the song?
USER
Sure, here are some lyries for the song:
Verse
You't the I've been waiting for,
My heart beats faster when I'm near you,
My love you will never fade.
Chorus:
You'l perfect, you' flawless,
SALMONN
Figure 20: Lyrics Writing
(It's soothing, elegant piano piece. At the end of the audio woman begins sing.)
Give me title of the music based on its rhythm and emotion.
USER
The music is beautiful piano piece with slow and emotional rhythm. It is a love song with a female vocalist singing in soft and gentle tone. The title of the music could be something like 'Forever Love' 'Endless Emotion'.
SALMONN
Figure 21: Titling Music
B PERFORMANCE ANALYSIS
As shown in Table 3, with activation tuning?, SALMONN
1. performs close to the state-of-the-art results the trained tasks including ASR, En2Zh, AAC and MC;
2. learns handle PR and OSR tasks which are difficult for the cascaded approach of "Vieuna
+ Whisper";
3. generalises well on a wide range of speech-grounder NLP tasks, especially such as En2De and En2Ja where SALMONN performs better than the "Whisper+Viuun' cascaded sys- tem, which indicates the superior of the audio-groundee avoiding eиror propagation and the loss of non-linguistic information (e.g. prosody);
system to cascade system on
4. tackles tasks such as audio-based storytelling and SAC which existing models can not han- dle to our best knowledge.
Added the requests of anonymous reviewers. We thank the reviewers for the suggestion.
19

---

Published as a conference paper at ICLR 2024
Table Mcan & standard deviation (in brackets) of the output sequence perplexity (PPL).
Model Checkpoint
ASR
AAC
PR
Story
SAC
Pre-training
(+000088)
5.2 (+0.422
270 (++60)
2.9 (+0.051)
5.4 (++0669
LoRA removed
5.9 (=1.))
40 (#2.0)
100 (=1.7)
2.2 (=0.028)
2.7 (#0.18)
Instruction Tuning
1.1 (=0.054)
5.0 (=0.56)
1.1 (#0.015)
2.4 (=0.030)
2.6 (#0.066)
LoRA removed
5.7
40 (13.3)
92 (+8.2)
2.2 (#0.023)
2.2 (#0.087)
Activation Tuning
1.1 (#0.048)
6.3 (=0.80)
1.1 (#0.015)
1.8 (#0.026)
1.4 (=0.0046)
The underlying reason for using the cascaded Whisper Vicuna system for reference values of the level 2 tasks lies in the fact that all level 2 tasks are zero-shot and there is no other audio- grounding system apart from SALMONN that can perform such tasks as zero-shot. An advantage of SALMONN over the cascaded system is that SALMONN can leverage both linguistic and par- alinguistic information required for spoken language understanding.
Despite such advantages, SALMONN has performance limitations some tasks.
1. First, PR is achieved by extending the LLM to consider phonemes new writing system. Since recognising phonemes requires finer- rrrergga modelling of pronunciation information than recognising the word pieces used by the original Whisper ASR, it is not easy for the SALMONN model built upon an existing Whisper speech encoder to perform as well as a specialised model the PR task.
2. Similarly, SALMONN has a relatively high WER on OSR since the Whisper ASR was not able to perform OSR.
3. The success of SQQA mainly relies the understanding of the spoken questions (e.g. "What the highest mountain in the world") and answering the questions based on the commonsense knowledge stored in the text-based L.M. The drop in SQQA performance indicates that the use of LoRA cross-modal adaptation may cause the L.M to "forget" some text-based commonsense knowledge.
C STATISTICS OF TASK OVER-FITTING
In this section, we analyse the changes of perplexity (PPL) in the three training stages to shed light on the underlying principle of activation tuning. As shown in Table 5, The PPI. of ASR and AAC tasks come very small values after the first pre-training stage, revealing that the model has learned cross-modal alignment. The PPL of PR drops after the instruction tuning stage since PR relies on LoRA to learn the output tokens of phonemes SAC also drop after instruction tuning, they are still not small enough for the tasks to be successfully performed, unless LoRA is removed test time Moreover, unlike the removal of LoRA, the PPLs on the ASR, AAC, and PR tasks remain almost unchanged after activation tuning, showcasing the advantages of this approach.
new "language". While the PPLs of Story and
additional activation tuning stage is performed.
D CHANGES IN PERPLEXITY DURING TRAINING
Changes in perplexity during the proposed three-stage training on six tasks including ASR, AAC, AST (En2Zh), OSR, Story, and SAC are visualized in Fig. 22.
E CALCULATION OF FOLLOWING RATE (FR) OR ACCURACY (ACC) FOR SQQA, SF, STORY AND SAC TASKS
For the SQQA and SF tasks, if the WER calculated between model output and the question in the speech is less than 30%, it is regarded that the model goes for ASR and does not follow instructions. For the Story task, we set the max length of output tokens to 200. Under this setting, answers shorter than 50 words regarded as disobeying the instructions, and count the number of different
20

---

Published as a conference paper at ICLR 2024
LotA wio L:RA
AV LoMA
wio LoRA
W/C LA
50
10
52.12
clst
CEst
(a) ASR
(b) Audio captioning.
(c) AST (En2Zh).
LArAA
53-16
59-
62-12
15
clpt
(d) Overlap speech recognition.
(e) Audio story telling.
(I Speech audio co-reasoning.
Figure 22: Changes in perplexity during the three-stage training. S1 is cross-modal pre-training
stage, S2 is instruction tuning stage and S3-t is the tth step in the activation tuning stage.
words in the story to represent the diversity metric for storytelling. For the SAC task, we use GPT- 3.5 to determine whether our model follows the instruction or answers the question correctly based on the background audio caption and the question.
F PROMPT TEMPLATE FOR SALMONN
To train SALMONN to generate responses to a text instruction given an audio input, we utilize prompt template follows:
USER: [Auditory Tokens] Text Prompt \n ASSISTANT:
The "[Auditory Tokens|" are the output variable length text-like tokens of the window-level Q- Former. Noted that this template matches the one used for training Vieuna (Chiang et al., 2023),
G PROMPTS INTERACTING WITH GPT3.5
In this work, utilize GPT3.5 to help us generate data and evaluate the quality of the model output automatically. We list our prompts for different purposes in Table 6. Words with all capital letters in the prompt will be replaced with the appropriate description before being fed into GPT3.5.
21

---

Published as a conference paper at ICLR 2024
H POTENTIAL TO EXTEND SALMONN TO SPEECH AND AUDIO GENERATION
Although SALMONN is designed focus on enabling LLMs with hearing abilities, it is possible to extend SALMONN to speech generation. 3 The human speech production mechanism related auditory perception. A well-known phenomenon attributed the speech chain is the "Lombard reflex" which describes the effect where individuals raise their voice level to be heard more clearly while speaking noisy environments (Lane & Tranel, 1971). This reveals the importance of having generic hearing abilities when empowering AI with fully buman-like speaking abilities, and the opportunities in text-to-Speech (TTS) synthesis enabled by SALMONN. This also matches the recent development in TTS that the text and audio contexts from the surrounding utterances are useful to achieve more natural prosody modelling and enable the use more natural and casual speech data (Xu et al., 2021; Guo et al., 2021; Oplustil-Gallegos al., 2021; Zhang al., 2023c).
Added the request of anonymous reviewer. We thank the reviewer for the suggestion.
22

---

Published as a conference paper at ICLR 2024
Purposes
Prompts
Below will give you some sentences that you will need to help generate **only one** question, and its corresponding swer. These sentences are captions some audio. Your question should be highly related to the audio caption, and your answer must be **correct** and should be simple and clear. 'nn Your re- sponse should strictly follow the format below: \n ("Question". "xxx", "Answer": "xxx") } in Here are the sentences:
To generate audio QA data given audio caption text.
Below will give you some sentences that you will need to help me generate **olly one** question, and its corresponding an- swer. Your question should be highly related to the sentences, and your answer must be *oorrct*** and should be simple and clear. \n Your response should strictly follow the format below: in ("Question": "xxx", "Answer": "xxx"] in Here are the sen- tences:
To generate speech QA data given speech recognition text.
Next will give you a question and give you the corresponding standard answer and the answer I said. You need to judge whether my answer is correct or not based on the standard answer to the question. I will give you the question and the corresponding swer in the following form: ("Question': 'xxx', 'Standard An- swer': 'xxx', 'My Answer': 'xxx") \n You need to judge the correctness of my answer, as well as state a short justification. Your responses need follow the Python dictionary format: \n {"Correct": True / False, "Reason": "xxx") } \n Now, I will give you the following question and answer: SENTENCEHERE \n Your response is:
To evaluate answers of the model of spoken-query- based question answering (SQQA).
There is audio clip, and there is person in the audio asking questions. now have an AI model that needs to go and an- swer the speaker's question based on the background audio. I'll tell you the question the speaker is asking the output of my Al model, and what you need to determine: whether my Al model is trying to answer the question and why. You need to be es- pecially careful that my model may just be describing the audio without hearing your question and answering it. You don't need to care about the correctness the answer. All you need to fo- cus on is whether the model is trying to answer the question. Your response needs to follow the format of the python dictio- nary: ("Response": "Yes/No", "Reason": "xxx"). An Question in audio: <QUESTION> An Model Output: <OUTPUT> \n Your Response:
To evaluate whether the model attempts do the speech audio co-reasoning (SAC) task.
There is audio clip, and there is person in the audio asking questions. now have an Al model that needs to go and answer the speaker's question based on the background audio. I'll tell you the question asked by the speaker, some description of the background audio, and the output of my AI model, and you need to decide whether my AI model answered it correctly, and why. Your response needs to follow the format of the python dictio- nary: ("Response": "Yes/No", "Reason": "xxx"). \ Question in
To evaluate whether the model successfully completes the SAC task.
audio: <QUESTION> \n Background Audio: <AUDIO> \n
Model Output: <OUTPUT> | Your Response:
Table 6: Purposes and prompts of using GPT3.5.
23


Reviews:
[{"review_id": "kYwKzroSmA", "reviewer": "Reviewer_KyXG", "summary": "This paper proposed SALMONN, which is a single unified multimodal model to integrate speech and audio encoders with a pre-trained text LLM. The paper shows that SALMONN can achieve competitive performance on a variety of speech tasks used in training, including ASR, ST, emotion recognition, audio QA, speaker verification, audio and music captioning etc. The paper also studies the capabilities of SALMONN on zero-shot capabilities such as ST on untrained languages, SLU, SQA, audio-based story telling, speech-audio co-reasoning etc. The paper also explores a new few-shot activation tuning approach.", "questions": "Please check the comments and concerns raised under Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)\tValue to the community: The proposed SALMONN model unifies modeling a wide variety of speech, audio, and music perception and understanding tasks into a single framework, which is a useful step towards the research for AGI. \n\n(2)\tThe innovation of the paper is mostly in choosing and piecing together existing approaches for this unified framework, using and evaluating a diverse set of speech/audio/music tasks for pre-training/instruction-finetuning, studying emergent capabilities, analyzing task-overfitting issue and exploring cheap activation tuning to alleviate catastrophic forgetting to training tasks.  SALMONN reasonably adopts several existing approaches. For speech and non-speech audio encoding, SALMONN integrates a speech encoder from Whisper and a BEATS audio encoder. SALMONN uses a Q-Former to convert encoder output to audio tokens for the text LLM (Vicuna in this work). LoRA is applied to align the augmented input space with output space to improve cross-modal alignment for the text LLM.  Following other works, SALMONN used a diverse set of speech, audio, and music tasks in pre-training and instruction finetuning of Q-Former and LoRA.  Notably,  this paper analyzed the task overfitting issue and provided insights for activation tuning to alleviate catastrophic forgetting from instruction tuning. The paper also studies the capabilities of SALMONN on handling cross-modal emergent tasks.  \n\n(3)\tOverall, the paper is clearly written.\n\n(4)\tEmpirical evaluations are comprehensive. The three levels are helpful organizations, as  speech tasks used in instruction tuning, unseen speech-based NLP tasks which can effectively evaluate speech-text cross-modal alignments, and the proposed new audio-based story telling and speech audio co-reasoning tasks which require understanding mixture of speech and non-speech auditory information.", "weaknesses": "(1)\tIn empirical validations, the choice of reference values (as shown in Table 2) needs to be clarified, and more importantly, these choices need to be justified. It is not clear which model size is used for Whisper when it is used as reference values. Also, the choice of simply cascading Whisper + Vicuna needs to be justified as the reference value for many tasks, since it may not be as competitive as other E2E models (e.g., recent speech LLMs), including SOTA. Without clear knowledge how strong these reference values are, it is not easy to judge how strong SALMONN performs as shown in Table 3.\n\n(2)\tSome key implementation details are missing.  The training data as shown in Table 1 are highly unevenly distributed. It is not clear methods such as data upsampling are used, or batches are designed for multi-task instruction fine-tuning.\n\n(3)\tThe paper focuses on general hearing capabilities of speech/audio/music. It would be useful to discuss how to extend the model to speech/audio/music generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3co9XfWL2J", "reviewer": "Reviewer_RbXH", "summary": "This paper presents SALMONN, a novel method for equipping LLMs with hearing abilities by leveraging additional adapter modules and LoRA weights to train the LLM on a range of speech, audio, and music understanding tasks.", "questions": "- What is \"monotonic alignment\" as mentioned in the Q-Former section?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The overall architecture design is quite clever, and its use of generally open source models and datasets is very important to the audio-text field as a whole.\n- The amount of time spent towards analyzing and mitigating the models failure modes is quite useful, and the authors provide an incredibly detail analysis of model behavior under different finetuning configurations.\n\nDespite some of the concerns mentioned below (which mostly address overall clarity rather than content), the paper presents an incredibly detailed analysis of a novel architecture and how to adapt LLMs for audio-based reasoning, and thus I recommend acceptance.", "weaknesses": "- The explanation of the pretraining stage could use a bit more depth. Namely, how is it that the Q-Former and LoRA weights are actually trained during the pretraining stage?\n- I think in general, the sections on task-overfitting and activation tuning are relatively hard to parse reading-wise and could be simplified. Unless I am misunderstanding something, task-overfitting is simply the idea that SALMONN overfits to the overrepresented tasks in the dataset, which I think the math in section 3.2 overcomplicates.\n- It is hard to tell in the ablations (5.2-5.4) what is being held fixed and what training configurations are being used. Namely, in section 5.2 are the results on the reduced LoRA scaling factor done with or without activation tuning?\n- The authors claim that the Level 3 tasks are harder directly, but given the myriad of evaluation metrics for each task it's hard to tell *why* these tasks are necessarily in their own class. Is there some way to show how these tasks are by nature an entirely more difficulty class of problems? Especially as SAC seems to be evaluated by ChatGPT outputs, it is hard to tell much about the actual performance of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCJIPxZI2w", "reviewer": "Reviewer_bK7i", "summary": "The authors propose to fuse both speech encoder (whisper) and general audio encoder (beats) as inputs, connect with LLM via a Q-Former and fine-tune with LoRA. Besides pre-training and instruction tuning stages, the authors also propose an activation tuning stage, which is to prevent from overfitting to short captions and is able to generate long and diverse stories. For some of the instruction tuning dataset, the authors leverage LLM for curation.", "questions": "- The process of activation tuning stage is not clear described, according to the last paragraph of 4.2, if the data is generated by SALMONN model, what do they look like? It would be helpful to provide an example. Also for teacher-forcing training, is it just a standard cross-entropy loss on generated text? How do you control the diversity and length of generated examples?\n- How are the instruction prompted for evaluation tasks? How do you instruct the model for evaluating these tasks?\n- Is ChatGPT also leveraged to generate text data for some evaluation tasks, especially for SAC and Story. If so, how are they leveraged to curate answers?\n- How are the prompted QA for training generated by ChatGPT verified?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The combination of speech and audio encoders are interesting ideas and the results of speech audio co-reasoning provide new capabilities for audio understanding.", "weaknesses": "- According to the results in table 3, it seems that the proposed method only works significantly better on level 3 tasks of Story and SAC, which the evaluation metrics are specifically designed and there is no other reference value from other models provided. It would be better to provide more information on the 2 tasks and providing other baseline performance on these two tasks.\n- For the Story task, it is worth including accuracy FR along with diversity FR. This can provide a more holistic understanding of the tradeoffs.\n- For some tasks in level 1 and 2, the performance of proposed method is significant worse, e.g. PR, OSR. and SQQA for level 2. It might worth providing some in depth discussion and analysis. For example, can it be that adding both whisper and beats features introduce more confusion to the model? How are the Q-former attending to the concatenated features to make predictions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)