Distilling an End-to-End Voice Assistant Without Instruction Training Data

## Abstract

Abstract Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT)
have led to models “forgetting” capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using > > > 100x less training compute.

## 1 Introduction

As Large Language Models (LLMs) capabilities increase, so does the value of bringing these capabilities to new modalities, including audio and speech (Shu et al., ; Wang et al., ; Gong et al., ).
Speech is a natural interaction surface for language technology (Murad et al., ), offering measurable efficiency gains for users (Ruan et al., ). One straightforward method of integrating speech with LLMs is to feed audio to an Automatic Speech Recognition (ASR) model and produce a text transcription for the LLM to use. However, this process loses meaningful information carried through tone, pacing, and accent (Upadhyay et al., ) regardless of the transcription accuracy. Furthermore, finetuning these pipelined systems requires supervision for both transcription and response generation, increasing annotation complexity and costs.

As such, LLMs that interface with speech directly have the potential to accelerate inference, reduce annotation costs, and capture the rich information inevitably lost by ASR. In this pursuit, a variety of works have trained audio-encoders on top of LLMs (Ao et al., ; Chen et al., ; Deshmukh et al., ; Chu et al., ; Wu et al., ), many of which utilize the same well-established approach: large-scale multi-task supervised finetuning (SFT).

Models using SFT face several challenges. First, without a large degree of task diversity in their training data, they often fail to generalize capabilities from the text-only LLM to speech. As observed in  Tang et al. (), freezing the weights of the text-only LLM is insufficient to prevent this “forgetting”. In order to generalize well, SFT must be trained with labeled data from a wide range of tasks and domains, with minimal imbalance between tasks. However, broad annotated speech instruction training data does not currently exist.

The limited instruction data that does exist is often collected from a small pool of speakers (Kim et al., ; Tomasello et al., ) or intended for evaluation rather than training (Faisal et al., ; Eisenstein et al., ). This lack of representation of speech from the wider population is likely to exacerbate biases in speech processing (Koenecke et al., ; Mengesha et al., ; Chan et al., ; Javed et al., ; Brewer et al., ). At present, Speech LLMs appear fundamentally limited by existing instruction data.

In this work, however, we argue that these “limitations” of existing data are artificially imposed by SFT. The speech community has already invested in large-scale data collection from the internet (Radford et al., ; Chen et al., ; Li et al., ), audiobooks (Panayotov et al., ; Pratap et al., ), and public archives (Galvez et al., ). Furthermore, several datasets have been explicitly gathered to represent diverse demographics (Porgali et al., ; Garg et al., ). However, these large-scale and diverse datasets are dominated by data in just one task: Automatic Speech Recognition (ASR). This means that models trained with SFT cannot make use of the entirety of this data without “forgetting” non-ASR capabilities.

We solve the “forgetting” problem by training a model that generalizes well despite using only ASR data. Rather than relying on external labels, our Distilled Voice Assistant (DiVA) self-supervises learning using the output distribution of an LLM in response to transcripts as a target, a cross-modal form of context distillation (Snell et al., ; Mu et al., ). We test our approach by training on just a single corpus, the CommonVoice, consisting of speech and transcriptions contributed by volunteers around the world and recorded on their own devices (Ardila et al., ).

Despite this data simplicity, DiVA generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, DiVA is preferred by users to our most competitive baseline Qwen 2 Audio in 72% of trials despite DiVA using over 100x less training compute. Beyond contributing a new Speech LLM, DiVA creates a new approach to Speech LLMs that trains more efficiently and generalizes better without requiring investment in new speech instruction data.

## 2 Related Work

**Table 1: High-Level comparison with state-of-the-art open-access Speech & Audio LLMs which we compare to. DiVA offers an entirely different approach to training using context distillation.**
| Model | Base LLM | Training Method | # Tasks | # Hours |
| --- | --- | --- | --- | --- |
| SALMONN | Alpaca | SFT | 12 | 4400 |
| Qwen Audio Chat | Qwen Chat | SFT | 31 | $\sim$50k |
| Qwen 2 Audio Instruct | Qwen2 Instruct | SFT, DPO | Unreported | $>$370k |
| DiVA (Ours) | Llama 3 | Distillation | N/A | 3.5k |

LLMs have been extended to both audio and image inputs using cross-modal encoders. For example, LLaVA (Liu et al., ) enables image understanding by connecting CLIP (Radford et al., ) to Vicuna (Chiang et al., ) through an MLP layer. Several recent works (Zhang et al., ; Gong et al., ; Tang et al., ; Chu et al., ; ) have connected audio-encoders (Gong et al., ; Hsu et al., ) to LLMs. There are two critical questions in this space.

How can audio features be transformed into a reasonable number of LLM input embeddings? Audio comes at high sample rates, and therefore, audio encoders often have a large number of outputs. For these features to be usable by the LLM, the dimensionality must be reduced, either by stacking consecutive features (Wu et al., ; Fathullah et al., ) or learning an adapter-module, such as an MLP (Liu et al., ; Gong et al., ), or Q-Former (Dai et al., ; Tang et al., ).

While learned approaches are more flexible, allowing for an adaptive reduction, they generally require learning a cross-attention mechanism, which generally requires significant training (Li et al., ). In this work, we find the best of both worlds by leveraging the Whisper decoder (Radford et al., ) to initialize the text-audio cross-attention mechanism of a Q-Former.

How can speech LLMs be trained to achieve instruction following abilities using existing data?
Prior work has explored two routes for creating instruction data without major financial investment. The first approach is to transform many existing datasets into an instruction-following format (Dai et al., ; Chu et al., ; Tang et al., ; Dai et al., ; Liu et al., ). In this case, limitations are often set by datasets that are not aligned with intended LLM usage and imbalances across tasks. The second approach is to train on synthetic responses to text representations of the new modality from commercial models (Liu et al., ; Gong et al., ).

Our approach is most related to the latter. Rather than generating data with an external model, we capitalize on the idea that instruction-tuned language models can provide valuable learning signals as long as the input is within the textual modality. Using the output distribution in response to text transcripts, we can more strongly guarantee the transfer of existing capabilities using context distillation (Snell et al., ; Mu et al., ).

How can we train foundation models for speech using open and permissively licensed data? Recently, frontier LLMs have begun integrating native speech capabilities. Unlike prior speech foundation models (Baevski et al., ; Hsu et al., ; Chen et al., ; Kim et al., ; Peng et al., ; ), these models offer virtual assistant capabilities rather than self-supervised audio representations or transcriptions. It is unclear to what degree these results are dependent on internal datasets, especially since even the state-of-the-art open-access Speech LLM with such capabilities reports no training data details other than size (Chu et al., ).

Similar to the Open Whisper-style Speech Model (OWSM) initiative (Peng et al., ), we use only open and permissively-licensed data. Furthermore, unlike the baselines we compare to in Table 1, we release the training code, rather than just the inference code, which can help reproduce a DiVA-style model easily.
Beyond our novel method, we believe this too broadens the ability to train and understand Speech LLMs.

## 3 Method

DiVA is an end-to-end voice and text assistant, trained using the process shown in Figure 1. We focus heavily on effectively using pretrained models in each domain. Similar to prior works, we initialize the audio encoder from the 1.5B parameter Whisper-Large-v3 model. Unlike prior works, we use all components of Whisper: not only reusing the encoder but also initializing a Q-Former from the decoder. We train this architecture using distillation loss on the input and output distribution of the text-only LLM, which we discuss in Section 3.2.

### 3.1 Model Initialization

When adding multimodal capabilities to an existing language model, the new modality must be represented as embeddings that can used in place of text token embeddings. Achieving this goal has two steps. First, meaningful features must be extracted from the input modality. Second, these features must be aggregated to be in-distribution for the downstream language model.

##### Audio Feature Extraction

We follow prior works (Chu et al., ; Tang et al., ) and use the Whisper encoder (Radford et al., ). Whisper first transforms the raw audio signal into a 128-channel time-frequency domain Mel-spectrogram. This is then passed through two 1D convolutions and used as embeddings fed to an unmodified Transformer architecture (Vaswani et al., ).

##### Audio-Text Feature Alignment

While the Whisper encoder extracts meaningful audio features, they are encoded at high granularity, with one token for every 40 milliseconds of input audio. By comparison, humans speak around one syllable every 150 milliseconds on average across languages (Coupé et al., ), and most tokens in an LLM vocabulary are made up of several syllables. This creates a mismatch between the granulatity between the Whisper encoder outputs and the downstream LLMs input distribution.

Prior work (Tang et al., ) addresses this using a Querying Transformer (Q-Former, Li et al. ), which learns static query embeddings with cross-attention to keys and values features from another modality. Given audio embeddings ${\bm{A}}$, the Q-Former learns a transformer with a cross attention mechanism $\sigma(\frac{{\bm{Q}}({\bm{K}}{\bm{A}}^{\intercal})}{\sqrt{d_{k}}})({\bm{V}}{
\bm{A}})$ where ${\bm{Q}}$ is a static set of query vectors, while ${\bm{K}}$ and ${\bm{V}}$ are projection matrices for the audio tokens. Conceptually, this cross-attention mechanism learns to dynamically aggregate information from the audio tokens into text-like tokens. This comes at the cost of significant training required to train the transformer from scratch.

The Whisper decoder, which prior work discards, is trained with a similar goal for ASR: mapping audio embeddings to discrete text tokens. Therefore, rather than learning Q-Former parameters from scratch, we initialize $K$ and $V$ from Whisper’s cross-attention mechanism. We adapt the model to a Q-Former by replacing the inputs with static query tokens $Q$. Finally, we project the output from the hidden dimension $h$ of Whisper to the hidden dimension $H$ of the LLM. This results in a set of $\{{\bm{t}}_{q}^{audio}\in\mathbb{R}^{H\times|Q|}\}$ output tokens representing the audio.

##### Text Decoding

For language processing and instruction following capabilities, we use Llama 3 (Dubey et al., )(^1^11Training was performed before the release of Llama 3.1.) and leave its weight frozen throughout training.

### 3.2 Distillation Losses

We optimize two loss functions based on audio recordings and corresponding text transcripts from ASR data. First, we minimize the distance between embeddings of audio and text on the input side of the LLM, similar to prior work for Vision-LMS (Radford et al., ; Li et al., ). Then, we minimize the KL Divergence between the output distribution in response to audio and text as a form of cross-modal context distillation (Mu et al., ; Snell et al., ).

#### 3.2.1 Cross-Modal Token Alignment Loss

To capture the mutual information between recordings and text transcripts, for a given ASR example (a text transcript and an audio recording), we align speech and text tokens as follows: The text transcript is embedded as $N$ text tokens ${\bm{t}}_{i}^{text}\in\mathbb{R}^{H\times N}$. The model produces $|Q|$ tokens from the audio recording where $|Q|>N$. We align these representations by minimizing the $L_{2}$ distance between the text embeddings and the final $i$ audio embeddings:

$$ $L_{con}=\sum_{n=0}^{N}|{\bm{t}}_{n}^{text}-{\bm{t}}_{Q-N+n}^{audio}|_{2}$ (1) $$

We use the final $N$ tokens of the audio embedding rather than the initial $N$ tokens due to the causal attention in Whisper’s decoder. Since the final tokens can attend to all preceding tokens, aligning the representations of the final tokens backpropagates signal to every token in the sequence. On the other hand, the additional $Q-N$ tokens provide information bandwidth for other information, such as sociophonetic cues, to be passed to the LLM.

Empirically, as we explore in Section 6, training with only token alignment leads to poor model quality, even when low loss is achieved. However, token alignment appears to enable reasoning between text and audio tokens, vastly improving text instruction adherence.

#### 3.2.2 Distillation from Output Embedding Distance

Voice assistant models should give coherent, helpful, and harmless responses to user speech. Thankfully, many openly accessible text-only LLMs have been extensively refined for these objectives. As such, our challenge is not to learn these behaviors but instead to transfer them to the audio modality. While, in theory, input token alignment could achieve this, even minor differences in input embeddings can significantly affect model behavior in practice (Cai et al., ).

Distillation loss, on the other hand, directly optimizes for the similarity of the output distribution (Hinton et al., ). Rather than distilling a large model into a smaller model, recent work has applied to distilling useful context into model weights, a process termed context distillation (Snell et al., ; Mu et al., ). Here, we apply context distillation across modalities, aiming to distill a text context into the audio modality under the assumption that the model should respond similarly to audio and text for most inputs.

In prior context distillation works, the full Kullback–Leibler (KL) Divergence has been shown to be prohibitively expensive at training time due to the large vocabulary of modern LLMs. Therefore, the KL Divergence is instead approximated by sampling random tokens (Snell et al., ). In our case, where the output embedding matrix is frozen, we show that there is an objective function easier to optimize:

###### Lemma 1 .

Given the probability $P_{t}$ from a teacher model and the probability $P_{s}$ from a student model, the KL Divergence is defined as $\mbox{KL}(P_{t},P_{s})=P_{t}\cdot(\log P_{t}-\log P_{s})$. For a transformer language model, $P_{s}=\sigma(O_{s}h_{s})$ where $h_{s}$ is the final hidden state, $O_{s}$ is the output embedding matrix, and $\sigma$ is the softmax function. Let $\theta_{s}$ be the student weights which we are trying to train to minimize the KL Divergence, then

$$ $\arg_{\theta_{s}}\min||{\bm{h}}_{s}-{\bm{h}}_{t}||_{2}\subset\arg_{\theta_{s}} \min\mbox{KL}(P_{t},P_{s})$ $$

###### Proof.

The KL divergence is minimized when $P_{s}=P_{t}$. Based on our definition of LM probability, this is equivalent to achieving $\sigma(O_{s}h_{s})=\sigma(O_{t}h_{t})$. In the special case we consider, where the teacher and student are initialized from the same weights, and $O_{s}$ is held constant, we know that $O_{s}=O_{t}$. Thus, a non-unique global minimum will be achieved when $h_{s}=h_{t}$, where the non-uniqueness comes from the softmax function $\sigma$, which is not injective.
∎

More importantly, we find that: (1) The gradient for L2 loss is much smoother than minimizing the KL divergence empirically(^2^22We explore this with an isolated small-scale experiment in Appendix A.2). (2) Since the vocabulary size of most modern LLMs is far larger than the hidden dimension, the distance between hidden states can be computed using far fewer operations than the KL divergence. In practice, we optimize the similarity of only the first predicted next token (after all $I$ text tokens/all $Q$ audio tokens) for efficiency, as Morris et al. () has shown that just a single token probability encodes significant information, both for prior and future tokens.

Notably, training with this loss only guarantees that the output distribution is well aligned in response to audio. However, our intuition is that this loss alone is likely to be less robust to input distribution shift without our token alignment loss, which we explore in Section 6.

###### Lemma 1 .

###### Proof.

## 4 Experimental Setup

### 4.1 Training Data

We utilize the English subsection of CommonVoice 17 (Ardila et al., ) as the dataset for all DiVA training runs. The dataset comprises just over 3.5 thousand hours of read text that has been crowdsourced and validated on the CommonVoice website. We select the CommonVoice for three reasons. Firstly, it is permissively licensed for commercial and research use. Secondly, it contains speech recorded in realistic settings on an individual’s device rather than in a professional studio. Finally, it includes speech from 93,725 speakers from a global pool of volunteers(^3^33Statistics drawn from the [official CommonVoice tracker](https://commonvoice.mozilla.org/en/languages)). The first factor means that the resulting DiVA models we release can be adopted for use broadly, while the latter two help make the training data more representative of real users.

### 4.2 Training Hyperparameters

We train for 4300 steps and a batch size of 512 using the AdamW Optimizer, a learning rate of 5E-5, and a weight decay of 0.1. This amounts to roughly two epochs over the data. We linearly warm up the learning rate for the first 1% of steps and then follow a cosine learning rate learning rate schedule which decays the learning rate to 0 over the course of the training run. The training run completes in approximately 12 hours on on a TPU v4-256 pod. (^4^44All training configurations can be found on [Github](https://github.com/Helw150/levanter/blob/ab863bb2e930d1fbb405681652cbcd75423325a3/config/via_commonvoice.yaml))

## 5 Quantitative and Qualitative Evaluations

We first assess how DiVA compares to baseline models for various spoken language benchmarks SFT models target. We evaluate on benchmarks for spoken question answering, speech classification, and speech translation. This provides a quantitative validation of DiVA’s generalization.

However, these benchmarks were all designed to test single task systems focused on each individual task. It is unclear whether these benchmarks capture the capabilities users expect from virtual assistants that speech LLMs are now powering commercially. To assess this, we run a further side-by-side comparison of DiVA with the best performing model on the benchmark evaluation Qwen 2 Audio (Chu et al., ).

##### Baselines

We compare our results to three openly available Speech Language Models: SALMONN, Qwen Audio Chat, and Qwen 2 Audio Instruct. Notably, all the baseline models utilize SFT covering these benchmark tasks. This makes them strong baselines: they all use similar scale base LLMs to DiVA, all make use of the Whisper encoder, and all have received direct supervision on the evaluated tasks. For our user study, we compare with Qwen 2 Audio, which reports state-of-the-art numbers and achieves the best average performance in our benchmarks.

Figure: Figure 2: Results across our two Question Answering benchmarks covering both standard evaluation and robustness to regional accents. Model correctness is assessed using the PANDA metric, which is tuned for strong correlation with human judgments of correctness (Li et al., ), and significance is from a paired bootstrap test (Dror et al., ).
Refer to caption: x2.png

### 5.1 Benchmarking

For question answering, we use HeySquad (Wu et al., ) and SDQA (Faisal et al., ), testing on 4,000 and 494 question-answer pairs, respectively. Classification is broken down into emotion recognition, sarcasm detection, and humor recognition. Emotion recognition is assessed on IEMOCAP (Busso et al., ) and MELD (Poria et al., ), with 1,241 and 2,608 utterances. We evaluate sarcasm detection on MUSTARD’s 690 clips (Castro et al., ) and humor recognition on URFunnyV2’s 7,614 examples. Finally, speech translation is tested on CoVoST 2, translating 15,500 English examples into seven commonly-tested typologically diverse languages (Clark et al., ). These datasets cover a wide range of traditionally benchmarked speech tasks drawn from prior work, which we cover in greater depth in Appendix A.3.

#### 5.1.1 Spoken Question Answering

We evaluate all models on zero-shot spoken question answering by prompting them with recorded audio of a speaker asking a question and the prompt: *You are a helpful assistant. Give answers as a simple single sentence*. The underlying LLMs for all baseline models are capable of question-answering, meaning that the audio encoder only needs to learn to map audio to the correct corresponding text to achieve strong results. This is a case where we expect DiVA to perform particularly well despite never having been explicitly trained on spoken questions.

Empirically, this expectation is met as shown in Figure 2. DiVA significantly (P$<$0.05) over the baselines by at least 10% (+5 PANDA) across both benchmarks and all accents.

However, it’s unclear whether lower accuracy can be directly attributable to “forgetting”. We qualitatively explore this question by labeling a sample of 50 responses from the HeySQUAD dataset for whether the responses include even an attempted answer relevant to the task.
Qwen Audio shows signs of severe forgetting, with 30% of responses ignoring the prompt instructions entirely and instead transcribing the question e.g. *”The citation for the Pearson v. Society of Sisters case is What is the citation for the Pearson v. Society of Sisters case?”*. By comparison, SALMONN, which takes inference time interventions to reduce overfitting by partially ablating the LoRA modules learned for the base LLM, sees reduced overfitting with only 8% of model responses ignoring the prompt and instead transcribing. Qwen 2 Audio sees further reduced overfitting, likely due to its DPO process using unreleased data, with only 4% instances where the instruction is ignored. DiVA, despite being trained only on transcription data, is the only model adheres to the instruction consistently.

#### 5.1.2 Speech Classification

One possible downside of our distillation approach is that the loss function contains minimal supervision for tasks where the audio of speech itself contains rich information through tone. However, tone is frequently correlated with the semantics of the text itself. We hypothesize this may enable the audio encoder to transfer some amount of this tone information based on weak supervision from text. To assess this, we evaluate on speech classification tasks where tone is likely to play a major role: Sarcasm Detection, Humor Detection, and Emotion Recognition.

Figure: Figure 3: Results across Emotion, Humor, and Sarcasm classification tasks. We measure class-weighted F1 for multi-class classification and accuracy for binary classification. Significance computed using a paired bootstrap test.
Refer to caption: x3.png

##### Emotion Recognition

For emotion classification, we prompt each model with the instructions *Respond in a single word what emotion the input exhibits. If there is no clear emotion, respond ’Neutral’.* To use each model as a classifier, we follow prior work (Hendrycks et al., ) and use the log-probabilities assigned to each possible label as classifier scores.

DiVA performs significantly better than both baselines on both the MELD benchmark, sourced from television, and IEMOCAPS, which operates over recorded conversations.
In comparison to DiVA, both baseline models struggle to predict a diverse array of labels. Qwen Audio predicts the emotion as Sadness for greater than 90% of inputs for both MELD and IEMOCAPS, while SALMONN and Qwen 2 Audio behaves similar with Neutral predictions.

These results are quite surprising given that DiVA is trained without explicit emotion supervision. However, many examples in both IEMOCAPS and MELD communicate emotion through both text and audio signals which may confound how well these evaluations capture true sociophonetic signal.

##### Sarcasm & Humor Detection

We also evaluate on two tasks where communicative intent is expressed largely through tone. For Sarcasm Detection, we prompt each model to *Respond whether the input is sarcastic. Answer with a simple yes or no* for sarcasm detection and *Respond whether the input is intended to be humorous. Answer with a simple yes or no*. In both cases, we compare the log-probability assigned to either the token ”Yes” or the token ”No.

No model performs particularly well in these tasks. None of the evaluated models perform significantly ($P>0.05$) better than chance on sarcasm detection and only Qwen Audio Chat performs better than chance on Humor Detection. This suggests there is significant progress to be made in enabling speech-oriented language models to understand more complex social signals in speech.

These tasks also highlight a shortcoming of distillation – namely, that DiVA inherits even non-desirable behaviors from the base LLM. Even when asked whether obviously very serious text is humorous or sarcastic, Llama 3 will almost always find a way to argue that it is humorous. DiVA inherits this behavior, predicting the ”Yes” label in both tasks over 90% of the time.

#### 5.1.3 Speech Translation

Figure: Figure 4: Results for Speech Translation across 7 typologically diverse languages. We evaluate using SacreBLEU and compute confidence intervals using a Paired Bootstrap.
Refer to caption: x4.png

Finally, we assess the speech-to-text translation capabilities of each model from English Speech to text in another language. We prompt each model with the instruction *Translate the input from {input_lang} to {output_lang}*.

Results on this benchmark are far more mixed. The original Qwen Audio performs best on Chinese and Japanese, Qwen 2 Audio performs best on Arabic, German and Indonesian, and DiVA performs best on Tamil and Turkish. Notably, the original Qwen trains with more than 3700 hours of speech-to-text translation data from CoVost2. While Qwen 2 does not report which tasks it trains on, it is likely it trains on similar or increased volumes of data from CoVost2 as the original Qwen. This highlights the data and compute efficient transfer of the DiVA approach, as both of these models trained on more translation specific data than DiVA used for it’s entire training.

DiVA’s most notable underperformance is in Chinese and Japanese, where it underperforms both other models. Inspecting DiVA’s outputs and comparing them to translations from Llama 3 in response to text, we again find that our distillation loss leads us to preserve a negative behavior — for both Chinese and Japanese, Llama 3 has a strong bias towards generating translations in the Latin alphabet (Pinyin and Romanji) rather than the expected native script. This leads to especially poor results in these languages.

Figure: Figure 5: Example of the double-blind interface for the user study with responses (Left: Qwen 2, Right: DiVA) to the speech *Can you tell me about Large Language Models in the style of a haiku?*.
Refer to caption: x5.png

### 5.2 Qualitative User Study

Finally, to get a sense of how well the resulting models match user preferences, we recruit participants to compare DiVA to the top performing baseline, Qwen 2 Audio.

#### 5.2.1 Recruitment & Study Design

We recruit 53 participants on the Prolific platform to provide preference ratings. Each user was allowed to contribute a maximum of 10 ratings, but able to opt-out at any time, resulting in 522 preference ratings comparing the models. We paid users 2.50$ per 10 ratings, which took fewer than 10 minutes of active time for all annotators involved, for an effective pay rate of 15$ per hour. We report annotator demographics in Appendix A.5.

We pre-screened for users who report familiarity with existing LLM chatbots and virtual assistants (e.g. ChatGPT, Gemini, Claude and others). In order to avoid biasing our participants, we prompt them without reference to specific tasks to *Record something you’d say to an AI Assistant! Think about what you usually use Siri, Google Assistant, or ChatGPT for*. Users were then shown responses from each model, without knowledge of which model was which. To avoid any positional bias, we shuffle the order which users were shown model responses for each recording submitted.

#### 5.2.2 Results

Despite no clear winner on all benchmarks for Qwen 2 and DiVA, DiVA generally is strongly preferred by users, with a 72% win rate at the preference level. At the user level, 41/53 (77%) of users preferred DiVA for the majority of their inputs. Beyond showing that DiVA improves preference alignment, this indicates that benchmarks may not correlate with practical usage.

## 6 Loss Ablation

Figure: Figure 7: Ablation of the loss components from Section 3. Distillation leads to a capable audio-only model, but token-alignment improves instruction adherence.
Refer to caption: x7.png

To better understand each component of our distillation loss, we investigate the influence of each loss component independently. In Figure 7, we compare results between the complete DiVA method, using just the output distillation loss, and using just the input token alignment loss.

##### Impacts of KL Divergence Loss

The most clear necessity for DiVA is the KL Divergence loss on the output distribution. Using token-alignment only does not simply lead to marginally worse results, it causes generations to be often incoherent. For generative tasks, the model often outputs sentences which are only vaguely semantically related to the input or unrelated markdown headers. In classification tasks, the token-alignment only model never performs significantly better than random guessing.

##### Impacts of Token Alignment Loss

This might raise the question: why use the token-alignment loss if it performs so poorly? In evaluations on question answering, this is certainly reasonable since using the KL Divergence loss alone already leads to stronger performance than the SFT baselines.

However, for translation and emotion recognition tasks, we see near-zero results from KL Divergence loss alone. Qualitatively, we observe that the distillation only model replies directly to the speech regardless of the text instructions.

We quantify this failure to adhere to instructions for the translation task using FastText Language ID (Joulin et al., ) on the outputs, under the assumption that outputs which are not in the correct target language are the result of ignored instructions. DiVA outputs the correct language 74% of the time while the distillation only model outputs the correct language only 1.4% of the time(^5^55We include LID results for all models in Appendix A.4).

This indicates that the token-alignment loss is key to achieving a model using distillation that can follow both audio instructions and text instructions such as a system prompt.

## 7 Conclusion

In summary, we release DiVA, an end-to-end Voice Assistant model capable of processing text and audio natively. Our cross-modal distillation loss from text to speech showcases a promising direction for cost-effective capabilities transfer from one modality to another. Our Distilled Voice Assistant generalizes to Spoken Question Answering, Classification, and Translation despite only being trained on transcription data. Furthermore, DiVA is preferred by users to our most competitive baseline Qwen 2 Audio in 72% of instances despite DiVA taking over 100x less training compute. Together, these contributions highlight a path forward for rapid adaptation of LLMs to Speech, without large investments in new training datasets(^6^66We include anonymized links to training and evaluation code in Appendix A.1).

## 8 Acknowledgements

The authors would like to thank Larry Heck, Karen Livescu, Ryan Li, Chenglei Si, Yijia Shao, and Rose Wang for their comments on this work and on audio model evaluation at various stages in this project. We also are very grateful for the code and system design review from David Hall when adding audio support into Levanter. Computing resources used for this work were funded through a Stanford HAI-GCP Cloud Credit Grant, as well as support from the Google TPU Research Cloud.

## 9 Contributions

Will and Diyi led the project, scoped the goals, and planned the overall experimental procedure. Will implemented and trained DiVA, as well as the inference code to serve interactive evaluations. Yanzhe helped Will design and validate the DiVA architecture and loss. Ella, Weiyan, and Michael helped format, integrate, and evaluate models on existing static benchmarks. All authors helped review, draft, and edit the writing of this work.

## Appendix A Appendix

### A.1 Reproducibility Statement

We release our [training code](https://github.com/Helw150/levanter/blob/will/distill/src/levanter/models/via.py), as well as [evaluation code, demo code & raw outputs](https://github.com/SALT-NLP/DiVA-Eval). All dataset processing details are included in Appendix A.3. We release all model weights, as well as inference code, for both ablations and the main model on [HuggingFace](https://huggingface.co/models?search=WillHeld/DiVA), where they have been downloaded $>$100,000 times externally since our public model release on July 26th, 2024 including for extensive external evaluations in English and Thai which concluded that *DiVA is the only model that performs well on the
Speech [Instruction Following] task, but it experiences a notable drop when tested
on Thai.* (Manakul et al., ).

### A.2 Toy Experiment on KL Divergence versus Hidden State Alignment

Beyond being a valid and efficient approximate of the KL Divergence, the $L_{2}$ loss should offer a more stable gradient, especially early in training when the output distributions are extremely different. When $P_{t}$ is positive and $P_{s}$ is near zero, the KL divergence explodes to extremely large values which can make optimization difficult and subject to significant numerical error.

In order to test this intuition, we set up a toy experiment where the student model outputs a single hidden state $h_{s}$ and the teacher model outputs a single hidden state $h_{t}$. In this highly simplified space, each model is fully parameterized by the these hidden states. We initialize and output vocabulary from the normal distribution with $32,000$ vocabulary items. Then, we optimize $h_{s}$ based on either the $L_{2}$ distance with $h_{t}$ or the KL Divergence with the output probabilities. Finally, for both procedures, we optimize for 100 steps with stochastic gradient descent, running the experiment 100 times at logarithmically increasing embedding dimensions, and plot the final KL divergence achieved under each loss function.

Figure: Figure 8: Empirical Comparison of the KL Divergence with our Proxy $L_{2}$ loss in a toy experimental setup. Optimizing the KL Divergence directly leads to worse KL Divergence than optimizing the $L_{2}$ loss. This gap increases as the hidden dimension becomes larger.
Refer to caption: x8.png

We see that, as the embedding dimension grows, optimizing the $L_{2}$ loss actually achieves lower KL divergence in this setup than optimizing the KL Divergence directly. To some extent, this makes sense as the $L_{2}$ loss is an incredibly simple convex function to optimize in this setting, while the KL divergence introduces significant additional complexity and a much sharper loss landscape early in optimization. We used this setup early in model design phases to help validate the choice of this approximation empirically, without training full scale models.

### A.3 In-Depth Evaluation Description

#### A.3.1 Spoken Question Answering

##### HeySquad

HeySquad Wu et al. () is a spoken question answering (QA) dataset that aims to measure the QA ability of digital agents. It is based on the SQuAD dataset Rajpurkar et al. () with 76K human-spoken and 97K machine-generated questions, and the corresponding answers. We evaluate the models on the open-source validation set with around 4K QA pairs.

##### Spoken Dialect Question Answering (SDQA)

SDQA Faisal et al. () assesses the robustness of Spoken Language Understanding to global phonological variation in English. The dataset is made up of the same 1000 questions spoken and recorded by speakers in 10 accent regions where English is frequently spoken. We evaluate on the 494 of these questions which contain ground truth answers.

#### A.3.2 Speech Classification

##### Emotion Recognition

Interactive Emotional Dyadic Motion Capture (IEMOCAP)
IEMOCAP Busso et al. () is a dataset of $\sim$12 hours of videos, audio, motion capture, and transcripts of actors performing both improvised and scripted scenes. The seven professional and three student actors perform emotionally expressive scenes. Each conversation turn in each scene was labeled by six evaluators as demonstrating “happiness,” “sadness,” “anger,” “surprise,” “fear,” “disgust,” “frustration,” “excitement,” “neutral state,” or “other.” We follow Yang et al. () and remove unbalanced class labels, resulting in 1241 audio utterances in the fifth fold used by  Tang et al. ().

Multimodal EmotionLines Dataset (MELD) MELD Poria et al. () contains 13,708 utterances labeled by emotion and collected from the sitcom Friends. MELD builds on EmotionLines Hsu et al. (); however, the authors of MELD ask annotators to watch the videos instead of simply reading the transcripts to produce labels. Three graduate student annotators labeled all utterances for emotions: “anger,” “disgust,” “fear,” “joy,” “neutral,” “sadness,” and “surprise,” as well as for sentiments “positive,” “negative,” “neutral.” We evaluate on the test set of 2608 utterances.

##### Communicative Intent Recognition

Multimodal Sarcasm Dataset (MUSTARD) MUSTARD Castro et al. () is a collection of 690 clips from the TV shows Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous, labeled as sarcastic or non-sarcastic by three annotators. The clips were collected primarily from YouTube using keywords like Chandler sarcasm, Friends sarcasm, etc. and sampled from MELD Poria et al. (). The final dataset was filtered to have an even number of labels of sarcastic and non-sarcastic clips. We evaluate on all 690 clips to test the models’ capability in understanding intended sarcasm.

URFunny URFunny (Hasan et al., ) is a multimodal humor recognition benchmark constructed from 90.23 hours of TED talk recordings, spanning 1741 speakers and 417 topics. TED produces transcripts for the talks, which contain ”[laughter]” markers that show when the audience laughs. The authors sampled the context and punchline before laughter markers for 8257 positive examples and random parts of the transcript without laughter markers for 8257 negative examples. URFunnyV2 filters out noise and reduces overlap in examples. We evaluate 7614 examples from the train split of URFunnyV2 to evaluate the models’ ability to understand speakers’ humorous intents.

#### A.3.3 Speech Translation

##### CoVoST 2

CoVoST 2 (Wang et al., ) is a speech-to-text translation benchmark to and from English. The speech inputs are sourced from the CommonVoice and professional translators are hired to translate the recording into a target language. The test dataset is large, made up of 15,500 examples translated from English to each target language. We evaluate on 7 target languages selected for their typological diversity in prior work (Clark et al., ).

### A.4 Language ID Outputs For All Models

**Table 2: Percentage of outputs for which Language ID matches the target language**
| Language | DiVA | KL Only | Token Alignment | Qwen | Qwen 2 | SALMONN |
| --- | --- | --- | --- | --- | --- | --- |
| Arabic | 84% | 2% | 0% | 95% | 90% | 19% |
| German | 90% | 1% | 0% | 99% | 98% | 77% |
| Indonesian | 85% | 1% | 0% | 97% | 97% | 77% |
| Japanese | 28% | 2% | 0% | 100% | 99% | 67% |
| Tamil | 96% | 1% | 0% | 60% | 79% | 8% |
| Turkish | 74% | 1% | 0% | 93% | 92% | 28% |
| Mandarin | 60% | 2% | 0% | 91% | 83% | 93% |

### A.5 Prolific User Demographics

**Table 3: Aggregate Metrics for Age, Gender Identity, and High-Level Ethnicity Information from our User Study. Our participants cover a wide range of ages, are gender balanced, and have a similar distribution of ethnicities as reported in the United States Census.**
| Age | Gender Identity | Ethnicity (Simplified) |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Median Age | 34 | Man | 50.9% | White | 59.2% |
| Max Age | 69 | Woman | 49.1% | Black | 16.3% |
| Minimum Age | 19 | Other | 0% | Asian | 12.2% |
|  |  |  |  | Mixed | 4.1% |
|  |  |  |  | Other | 8.2% |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Have you compared your approach with an ASR -> LLM based pipeline? One would imagine it would score poorly on the emotion recognition tasks, but probably very high on others.
- Have you considered diving deeper into what kind of data is requires to achieve specific results? For example, you could ablate the amount of commonvoice data required to achieve specific results; explore other datasets and/or languages and measure their impact on downstream tasks. This would provide a valuable contribution to the community.
soundness: 3
strengths: - Interesting approach which takes several pre-trained models (Llama, Whisper), connects them via a q-former based adapter and trains a small portion of the combined model to respond to audio inputs.
- Scores relatively well on various benchmarks compared to some well known models.
- The main contribution is showing that it is sufficient to teach an existing instruction-tuned model to understand audio through relatively light weight techniques which then allows the model to follow instructions via audio queries.
summary: This paper proposes DiVA, a voice assistant model that is able to follow both spoken and written instructions. It is trained via a dual distillation and alignment loss and shows relatively strong results on various benchmarks including head to head comparison with Qwen 2 Audio. The authors propose a "q-former" style injection of audio into the model, which is initialized from a Whiper model, and a text model initialized with Llama weights and left frozen which consumes the audio input. This model is then trained for relatively small amount of steps on CommonVoice data to learn input alignment as well as output alignment (as compared to text labels).
weaknesses: There is not a lot of novelty in this method besides choosing which data to train on. Q-former or in general attention based pooling (e.g. as in perceiver) is well known; L2 loss as a replacement for KLD has also been around for awhile (e.g. in Soundnet: Learning sound representations from unlabeled video.). Putting some of these pieces together to allow Llama models to process audio input has also been explored, for example in Nvidia's SpeechLLM or SALMONN (which is cited in this paper and apparently does not use Llama). In my opinion demonstrating that full SFT is not required to obtain an audio understanding model is interesting but not sufficient.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. How is speech generated from the model output?
2. How is the quality of the output speech compared with the ground truth speech? 
3. it is stated that without the KL-divergence loss the speech output is incoherent, how was this measured?
4. Are there any audio samples from the model?
5. In what voices is the model able to generate speech?
soundness: 3
strengths: 1. The proposed method is novel and easy to reproduce.
2. Evaluations show strong results on a wide variety of tasks. 
3. The paper is clearly written. 
4. Authors use publicly available datasets and fine-tuned models.
summary: This paper proposes a method named DiVA for training a speechLLM without using any instruction data in the speech modality. To perform this, an LLM is adapted to enable speech input and output by first combining it with a Whisper based speech encoder, and then using two Novel loss term to train the new modules. The two losses are a cross-model alignment loss which aims to alight the speech and text modalities, and a (simplified) KL divergence loss on the distribution of text and speech outputs. The authors evaluate the paper on 3 groups of tasks, which are spoken question answering, speech classification and speech to speech translation. In all 3 task groups the method reaches state of the art results and surpassed the baseline methods. 

Overall, this paper shows a significant contribution and I recommend to accept it.
weaknesses: 1. It is unclear how speech (waveform) is generated from the model output.
2. The paper is missing some direct evaluation of the speech quality, such as MOS experiments. 
3. For the speech to speech translation task, the paper is lacking an evaluation of the speaker similarity and translation quality (ASR-BLEU) metrics, this may better inform the reader about the overall quality of the translations. 
4. While the paper claims to have created a voice assistant, there is no evaluation or clarification what a voice assistant means. It isn't clear that good performance on the tasks stated in the papers results in a voice assistant. This should be clarified or rephrase.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: Questions:
1. In Section 3.1, the authors state that they use the decoder weights of Whisper to initialize the Q-Former’s _cross attention_. However, in Section 3.2.1, it’s mentioned that the speech tokens are processed with _causal attention_ in Whisper's decoder. This seems contradictory and is confusing. Could the author clarify the architecture of the Q-Former adapter? 
2. In Section 3.2.1, the cross-modal token alignment loss is applied only to the last $N$ tokens. However, this choice is unclear since Q-Former’s output tokens lack causal relationships. Additionally, the number of speech tokens $Q$ is defined as a hyperparameter, so the assumption that $Q>N$ may not hold for all inputs during inference. Could the author provide further justification for these design choices and proofs for the assumption about token counts?
3. In Section 3.2.1, the author claim that the additional $Q-N$ token provide information bandwidth for other information. However, I see no evidence to support this claim. Is there any statistic on the number of tokens that did not undergo alignment during training? If so, how does dropping these tokens affect the model's performance? 
4. In Section 3.2.2, the authors claim that the proposed $L_2$ loss can be computed more efficiently than KL divergence. Could additional evidence on training costs (e.g., computation time or resource usage) be provided to support this claim?
5. The ablation study indicates that the token alignment loss aids the model in adhering to text instructions. However, this difficulty in following text instructions might stem from DiVA being trained solely on speech inputs without accompanying text instructions. Could incorporating text instructions before speech tokens during training improve performance? Furthermore, if this modification were implemented, would the token alignment loss still be necessary, or could it be reduced or omitted?

Typos and minor mistakes:
1. In Table 1, the base LLM for Qwen 2 Audio Instruct is listed as Qwen2 Instruct. However, according to the technical report, the correct base model should be Qwen-7B. 
2. In Equation (1), the subscripts in the summation notation should start from 1.
3. In Section 3.2 line 247, "L2" should be $L_2$

[1] Chu Y, Xu J, Yang Q, et al. Qwen2-audio technical report[J]. arXiv preprint arXiv:2407.10759, 2024.
soundness: 3
strengths: 1.  The cross-modal alignment approach effectively retains general instruction-following abilities, addressing the common issue of forgetting in supervised fine-tuning (SFT).
2.  The Distilled Voice Assistant (DiVA) surpasses previous SOTA models, such as Qwen2-Audio, with significantly lower resource requirements.
3.  A qualitative user study shows DiVA’s strong alignment with human preferences in conversational quality.
4.  Ablation analysis confirms the distinct contributions of cross-modal token alignment and embedding distillation loss to DiVA’s performance.
summary: This paper proposes a novel approach for training Speech LLMs in the absence of explicit instruction-following data. The authors introduce a method to transfer the instruction-following and conversational capabilities of text-based LLMs to speech-based models by leveraging only ASR data. Specifically, the proposed approach aligns input tokens using a cross-modal token alignment loss and output representations via embedding distillation loss, effectively bridging the modality gap between speech and text. Experimental results demonstrate that this method generalizes well to downstream tasks, including spoken QA and speech translation.
weaknesses: 1.  **Limited Novelty:** Although the authors suggest an "alternative paradigm" for aligning speech inputs to text-based LLMs, similar approaches involving distillation text responses for cross-modal alignment have already been explored in works like BLSP [1] and AudioChatLlama [2]. A more detailed discussion and comparison with these works would help clarify the unique contributions. 
2.  **Potential Limitations of Q-Former as Modality Adapter:** DiVA employs a Q-Former as the modality adapter to convert Whisper outputs into tokens. However, recent research in Vision-Language Models suggests that Q-Former can introduce semantic deficiency and "redundant double abstraction," making it less effective than simpler alternatives like MLP with average pooling [3]. Furthermore, the fixed number of query tokens restricts the model’s ability to process speech of varying lengths, potentially limiting its adaptability.
3.  **Unclear Basis for Performance Gains:** The claim that DiVA outperforms SOTA models like Qwen2-Audio may be misleading, given that DiVA builds on Llama-3—a stronger backbone than Qwen2-Audio’s Qwen-7B. This discrepancy makes it difficult to attribute performance gains solely to the proposed training method. Including the text-only performance of the backbone models (Llama-3 / Qwen-7B) or training DiVA on the same backbone as Qwen2-Audio could clarify the source of the observed improvements. 



[1] Wang C, Liao M, Huang Z, et al. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing[J]. arXiv preprint arXiv:2309.00916, 2023.

[2] Fathullah Y, Wu C, Lakomkin E, et al. AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 5522-5532.

[3] Yao L, Li L, Ren S, et al. DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models[J]. arXiv preprint arXiv:2405.20985, 2024.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - The authors should clarify the differences between this model and an ASR-LLM cascade, as mapping speech to text-like tokens questions both novelty and latency gains.
- Single-token predictions seem insufficient to capture temporal dynamics, potentially reducing DiVA’s effectiveness for tasks that require detailed temporal context.
soundness: 2
strengths: - The use of ASR data alone to improve instruction-following behavior could offer cost benefits by reducing dependence on annotated instruction data.
- DiVA’s evaluation across diverse benchmarks provides a quantitative assessment of the model’s instruction-following performance.
summary: This paper proposes the Distilled Voice Assistant (DiVA), a Speech Large Language Model (LLM) trained on ASR data alone to enhance instruction-following abilities through self-supervised distillation. The authors align DiVA’s responses with those of a text-only LLM to enable cross-modal transfer of instruction adherence. However, the approach does not present substantive methodological innovation, as it closely mirrors existing distillation and instruction-following techniques. Furthermore, the work critically lacks comparisons to similar method in this domain, failing to substantiate its claimed improvements over key baseline methods.
weaknesses: - Lack of Novelty: The paper’s distillation approach lacks originality and does not go beyond established self-supervised and cross-modal transfer methods. Its reliance on prior distillation techniques without meaningful innovation limits the work's impact.
- Missing Comparisons to Key Similar Work: The paper does not compare DiVA to highly relevant models, such as those using cross-entropy on target sequences or alternative distillation methods, failing to clarify any advantages over standard methods in the field [1, 2, 3].
- Insufficient Literature Survey: The paper omits numerous relevant works, resulting in a limited survey that neglects essential context for DiVA’s approach within the broader Speech LLM and instruction-following literature.
- Paralinguistic Claims: Assertions about capturing paralinguistic features (e.g., sarcasm, emotion) are questionable given that DiVA’s speech embeddings are mapped to text-like embeddings, likely relying more on text semantics than true paralinguistic cues.



1. Fathullah, Yassir, et al. "AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
2.  Wang, Chen, et al. "Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing." arXiv preprint arXiv:2309.00916 (2023).
3. Wang, Chen, et al. "BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillation." arXiv preprint arXiv:2405.19041 (2024).

Meta-review: The paper introduces DiVA (Distilled Voice Assistant), a model for training speech-based large language models (Speech LLMs) without requiring explicit instruction-following data in the speech modality. The key components of the approach are: 1) integrating a Whisper-based speech encoder with a text model initialized from Llama weights, allowing the model to process both speech and text inputs. The text model remains frozen while the combined model is trained on a relatively small amount of data, including CommonVoice. 2) training the model using dual distillation and alignment loss techniques, with cross-modal alignment loss to align speech and text modalities and embedding distillation loss to align output representations of speech and text. With these components, DIVA demonstrates competitive performance across several task groups, including spoken question answering, speech classification, and speech-to-speech translation, and surpassing baseline methods, such as Qwen2-Audio.

Strength of this paper

- Competitive performance: DiVA surpasses previous SOTA models, like Qwen2-Audio, with significantly lower resource requirements, making it a more resource-efficient solution. Evaluations show competitive results across various tasks and benchmarks, with DiVA performing well in instruction-following and cross-modal alignment. These experiment results support the model’s efficacy.
- A cost-effective methodology of using ASR data alone to improve instruction-following behavior and reducing reliance on annotated instruction data. The paper also demonstrates that it is sufficient to teach an existing instruction-tuned model to understand audio using relatively lightweight techniques, effectively enabling the model to follow instructions via audio queries.
- The paper is clearly written, and it uses publicly available datasets and fine-tuned models, which increases the paper's accessibility and reproducibility.


Weakness of this paper

- Limited Novelty: the method lacks novelty, as many components (e.g., attention-based pooling like Q-former, L2 loss as a replacement for KLD) are well-known in existing literature. Combining these pieces to process audio input with Llama models, though interesting, has been explored in previous works like Nvidia's SpeechLLM and SALMONN, which also use similar strategies. The proposed distillation approach, which relies on self-supervised and cross-modal transfer methods, does not go beyond well-established techniques and lacks meaningful innovation, limiting its impact.
- Unclear Model Outputs and Evaluation: The paper does not clarify how speech (waveform) is generated from the model output, leaving a gap in understanding how the system produces audible speech. There is no direct evaluation of speech quality, such as MOS (Mean Opinion Score) experiments, which would be valuable to assess the quality of the generated speech. The speaker similarity and translation quality (ASR-BLEU) metrics are missing for the speech-to-speech translation task. These would help evaluate the overall quality of the translations and speaker consistency. Besides, The evaluation of the voice assistant claims is unclear. The paper mentions the creation of a voice assistant but does not explain or evaluate what this entails, raising questions about the paper's practical applications and performance in real-world tasks.
- Lack of Comprehensive Comparisons: The paper does not compare the method with important works, such as those using cross-entropy or other distillation methods (e.g., BLSP, AudioChatLlama), which would clarify the advantages and uniqueness of the proposed approach. A more detailed comparison with existing methods like Qwen2-Audio and the Llama-3 backbone is needed, as claims about DiVA outperforming state-of-the-art models may be misleading due to discrepancies in model architectures and backbones.
- Lastly, related work and references is not incomplete, e.g., omitting numerous relevant works in the Speech LLM and instruction-following domains. This makes it harder to understand the position of this work, and what's the trade-offs and how proposed work compared with existing approaches, to help readers better understand the method's limitations and future applicability.

Decision: Reject