Distilling an End-to-End Voice Assistant Without Instruction Training Data

Authors: William Barr Held, Minzhi Li, Michael J Ryan, Weiyan Shi, Yanzhe Zhang, Diyi Yang
Primary area: foundation or frontier models, including LLMs
Keywords: Multi-Modal LLMs, Voice Assistants, Distillation

Abstract:
Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) 
 have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72\% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using $>$100x less training compute.

Decision: reject
Reviewer scores: 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729934687659

thread_type:
official_review

<thread_object>
creator: reviewer_wqdk

<content>
summary:
This paper proposes the Distilled Voice Assistant (DiVA), a Speech Large Language Model (LLM) trained on ASR data alone to enhance instruction-following abilities through self-supervised distillation. The authors align DiVA’s responses with those of a text-only LLM to enable cross-modal transfer of instruction adherence. However, the approach does not present substantive methodological innovation, as it closely mirrors existing distillation and instruction-following techniques. Furthermore, the work critically lacks comparisons to similar method in this domain, failing to substantiate its claimed improvements over key baseline methods.

strengths:
- The use of ASR data alone to improve instruction-following behavior could offer cost benefits by reducing dependence on annotated instruction data.
- DiVA’s evaluation across diverse benchmarks provides a quantitative assessment of the model’s instruction-following performance.

weaknesses:
- Lack of Novelty: The paper’s distillation approach lacks originality and does not go beyond established self-supervised and cross-modal transfer methods. Its reliance on prior distillation techniques without meaningful innovation limits the work's impact.
- Missing Comparisons to Key Similar Work: The paper does not compare DiVA to highly relevant models, such as those using cross-entropy on target sequences or alternative distillation methods, failing to clarify any advantages over standard methods in the field [1, 2, 3].
- Insufficient Literature Survey: The paper omits numerous relevant works, resulting in a limited survey that neglects essential context for DiVA’s approach within the broader Speech LLM and instruction-following literature.
- Paralinguistic Claims: Assertions about capturing paralinguistic features (e.g., sarcasm, emotion) are questionable given that DiVA’s speech embeddings are mapped to text-like embeddings, likely relying more on text semantics than true paralinguistic cues.



1. Fathullah, Yassir, et al. "AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
2.  Wang, Chen, et al. "Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing." arXiv preprint arXiv:2309.00916 (2023).
3. Wang, Chen, et al. "BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillation." arXiv preprint arXiv:2405.19041 (2024).

questions:
- The authors should clarify the differences between this model and an ASR-LLM cascade, as mapping speech to text-like tokens questions both novelty and latency gains.
- Single-token predictions seem insufficient to capture temporal dynamics, potentially reducing DiVA’s effectiveness for tasks that require detailed temporal context.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730042028714

thread_type:
official_review

<thread_object>
creator: reviewer_ukgu

<content>
summary:
This paper proposes a novel approach for training Speech LLMs in the absence of explicit instruction-following data. The authors introduce a method to transfer the instruction-following and conversational capabilities of text-based LLMs to speech-based models by leveraging only ASR data. Specifically, the proposed approach aligns input tokens using a cross-modal token alignment loss and output representations via embedding distillation loss, effectively bridging the modality gap between speech and text. Experimental results demonstrate that this method generalizes well to downstream tasks, including spoken QA and speech translation.

strengths:
1.  The cross-modal alignment approach effectively retains general instruction-following abilities, addressing the common issue of forgetting in supervised fine-tuning (SFT).
2.  The Distilled Voice Assistant (DiVA) surpasses previous SOTA models, such as Qwen2-Audio, with significantly lower resource requirements.
3.  A qualitative user study shows DiVA’s strong alignment with human preferences in conversational quality.
4.  Ablation analysis confirms the distinct contributions of cross-modal token alignment and embedding distillation loss to DiVA’s performance.

weaknesses:
1.  **Limited Novelty:** Although the authors suggest an "alternative paradigm" for aligning speech inputs to text-based LLMs, similar approaches involving distillation text responses for cross-modal alignment have already been explored in works like BLSP [1] and AudioChatLlama [2]. A more detailed discussion and comparison with these works would help clarify the unique contributions. 
2.  **Potential Limitations of Q-Former as Modality Adapter:** DiVA employs a Q-Former as the modality adapter to convert Whisper outputs into tokens. However, recent research in Vision-Language Models suggests that Q-Former can introduce semantic deficiency and "redundant double abstraction," making it less effective than simpler alternatives like MLP with average pooling [3]. Furthermore, the fixed number of query tokens restricts the model’s ability to process speech of varying lengths, potentially limiting its adaptability.
3.  **Unclear Basis for Performance Gains:** The claim that DiVA outperforms SOTA models like Qwen2-Audio may be misleading, given that DiVA builds on Llama-3—a stronger backbone than Qwen2-Audio’s Qwen-7B. This discrepancy makes it difficult to attribute performance gains solely to the proposed training method. Including the text-only performance of the backbone models (Llama-3 / Qwen-7B) or training DiVA on the same backbone as Qwen2-Audio could clarify the source of the observed improvements. 



[1] Wang C, Liao M, Huang Z, et al. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writing[J]. arXiv preprint arXiv:2309.00916, 2023.

[2] Fathullah Y, Wu C, Lakomkin E, et al. AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 5522-5532.

[3] Yao L, Li L, Ren S, et al. DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models[J]. arXiv preprint arXiv:2405.20985, 2024.

questions:
Questions:
1. In Section 3.1, the authors state that they use the decoder weights of Whisper to initialize the Q-Former’s _cross attention_. However, in Section 3.2.1, it’s mentioned that the speech tokens are processed with _causal attention_ in Whisper's decoder. This seems contradictory and is confusing. Could the author clarify the architecture of the Q-Former adapter? 
2. In Section 3.2.1, the cross-modal token alignment loss is applied only to the last $N$ tokens. However, this choice is unclear since Q-Former’s output tokens lack causal relationships. Additionally, the number of speech tokens $Q$ is defined as a hyperparameter, so the assumption that $Q>N$ may not hold for all inputs during inference. Could the author provide further justification for these design choices and proofs for the assumption about token counts?
3. In Section 3.2.1, the author claim that the additional $Q-N$ token provide information bandwidth for other information. However, I see no evidence to support this claim. Is there any statistic on the number of tokens that did not undergo alignment during training? If so, how does dropping these tokens affect the model's performance? 
4. In Section 3.2.2, the authors claim that the proposed $L_2$ loss can be computed more efficiently than KL divergence. Could additional evidence on training costs (e.g., computation time or resource usage) be provided to support this claim?
5. The ablation study indicates that the token alignment loss aids the model in adhering to text instructions. However, this difficulty in following text instructions might stem from DiVA being trained solely on speech inputs without accompanying text instructions. Could incorporating text instructions before speech tokens during training improve performance? Furthermore, if this modification were implemented, would the token alignment loss still be necessary, or could it be reduced or omitted?

Typos and minor mistakes:
1. In Table 1, the base LLM for Qwen 2 Audio Instruct is listed as Qwen2 Instruct. However, according to the technical report, the correct base model should be Qwen-7B. 
2. In Equation (1), the subscripts in the summation notation should start from 1.
3. In Section 3.2 line 247, "L2" should be $L_2$

[1] Chu Y, Xu J, Yang Q, et al. Qwen2-audio technical report[J]. arXiv preprint arXiv:2407.10759, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730042815653

thread_type:
official_review

<thread_object>
creator: reviewer_rrhc

<content>
summary:
This paper proposes a method named DiVA for training a speechLLM without using any instruction data in the speech modality. To perform this, an LLM is adapted to enable speech input and output by first combining it with a Whisper based speech encoder, and then using two Novel loss term to train the new modules. The two losses are a cross-model alignment loss which aims to alight the speech and text modalities, and a (simplified) KL divergence loss on the distribution of text and speech outputs. The authors evaluate the paper on 3 groups of tasks, which are spoken question answering, speech classification and speech to speech translation. In all 3 task groups the method reaches state of the art results and surpassed the baseline methods. 

Overall, this paper shows a significant contribution and I recommend to accept it.

strengths:
1. The proposed method is novel and easy to reproduce.
2. Evaluations show strong results on a wide variety of tasks. 
3. The paper is clearly written. 
4. Authors use publicly available datasets and fine-tuned models.

weaknesses:
1. It is unclear how speech (waveform) is generated from the model output.
2. The paper is missing some direct evaluation of the speech quality, such as MOS experiments. 
3. For the speech to speech translation task, the paper is lacking an evaluation of the speaker similarity and translation quality (ASR-BLEU) metrics, this may better inform the reader about the overall quality of the translations. 
4. While the paper claims to have created a voice assistant, there is no evaluation or clarification what a voice assistant means. It isn't clear that good performance on the tasks stated in the papers results in a voice assistant. This should be clarified or rephrase.

questions:
1. How is speech generated from the model output?
2. How is the quality of the output speech compared with the ground truth speech? 
3. it is stated that without the KL-divergence loss the speech output is incoherent, how was this measured?
4. Are there any audio samples from the model?
5. In what voices is the model able to generate speech?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730855586318

thread_type:
official_review

<thread_object>
creator: reviewer_rub3

<content>
summary:
This paper proposes DiVA, a voice assistant model that is able to follow both spoken and written instructions. It is trained via a dual distillation and alignment loss and shows relatively strong results on various benchmarks including head to head comparison with Qwen 2 Audio. The authors propose a "q-former" style injection of audio into the model, which is initialized from a Whiper model, and a text model initialized with Llama weights and left frozen which consumes the audio input. This model is then trained for relatively small amount of steps on CommonVoice data to learn input alignment as well as output alignment (as compared to text labels).

strengths:
- Interesting approach which takes several pre-trained models (Llama, Whisper), connects them via a q-former based adapter and trains a small portion of the combined model to respond to audio inputs.
- Scores relatively well on various benchmarks compared to some well known models.
- The main contribution is showing that it is sufficient to teach an existing instruction-tuned model to understand audio through relatively light weight techniques which then allows the model to follow instructions via audio queries.

weaknesses:
There is not a lot of novelty in this method besides choosing which data to train on. Q-former or in general attention based pooling (e.g. as in perceiver) is well known; L2 loss as a replacement for KLD has also been around for awhile (e.g. in Soundnet: Learning sound representations from unlabeled video.). Putting some of these pieces together to allow Llama models to process audio input has also been explored, for example in Nvidia's SpeechLLM or SALMONN (which is cited in this paper and apparently does not use Llama). In my opinion demonstrating that full SFT is not required to obtain an audio understanding model is interesting but not sufficient.

questions:
- Have you compared your approach with an ASR -> LLM based pipeline? One would imagine it would score poorly on the emotion recognition tasks, but probably very high on others.
- Have you considered diving deeper into what kind of data is requires to achieve specific results? For example, you could ablate the amount of commonvoice data required to achieve specific results; explore other datasets and/or languages and measure their impact on downstream tasks. This would provide a valuable contribution to the community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731694611944

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
On Request for Additional Comparisons to Other Models

comment:
We thank the reviewers for their comments! We are working hard to provide substantive responses backed with experimental results to address reviewer concerns wherever possible. We will continue to provide concrete responses to these throughout the discussion period! The first of these empirical results is in regards to the reviewer request for further baseline comparisons to other recent Speech LLMS. Where possible we have run additional evaluations to directly address this feedback! 

First, we focus on specific points surrounding particular models that the reviewers have requested comparisons to. These models are BLSP, BLSP-KD, and AudioChatLlama: 

1. We have run our evaluations on BLSP, which performs hard-distillation on the outputs using cross-entropy loss, showing that DiVA improves performance across all metrics (see Table below) compared with this model. 
| Task        | DiVA   | BLSP   |
| ----------- | ------ | ------ |
| SDQA-Mean   | **53.58%** | 44.03% |
| HeySquad    | **55.17%** | 46.78% |
| IEMOCAPS    | **50.60**  | 42.88  |
| MELD        | **41.34**  | 40.12  |
| UrFunny     | **50.24%** | 49.92% |
| MUSTARD     | **52.61%** | 48.41% |
| COVOST-Mean | **13.80**  | 5.05   |

2. Neither BLSP-KD (published on Arxiv on May 29, 2024) and AudioChatLlama (published in NAACL on June 15, 2024) release their model weights or the code used to produce these models. We have emailed the authors of these works to see if we can get access to perform these comparisons. The BLSP-KD author unfortunately said that they cannot release their model as it was produced during a company internship. We are still waiting on a reply front the AudioChat Llama author to reply and will respond further when they do.
\
Without code or model weights it is difficult to compare directly to these recently released works, highlighting the importance of our entirely open-source pipeline for this work. However, we have done our best to provide empirical comparisons to these works based on available information in the next two points.

3. Our "Distillation Only" ablation closely mirrors AudioChatLlama's described methodology, which performs distillation on outputs only, revealing important limitations in generalization. Specifically, while this approach proves effective for Spoken Question-Answering (as both our works show), our broader evaluation reveals that it fails to generalize to tasks requiring text instruction following. This highlights the importance of our usage of both input and output distribution alignment.

4. Without access to the particular model, we instead compare the results from our model on the same COVOST languages they evaluate on in BLSP-KD to the results they report in their paper. We show this comparison below, where DiVA outperforms BLSP-KD on 5/7 languages used in the BLSP-KD paper. 

| Model              | en-ca | en-de | en-id | en-ja | en-sl | en-sv | en-zh |
| ------------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
| BLSP-KD (Reported) | 23.5  | 24.4  | 13.4  | **21.3**  | 10.7  | 24.5  | **41.3**  |
| DiVA               | **29.07** | **27.6**  | **22.8**  | 6.2   | **11.2**  | **24.7**  | 12.2  |

As noted in our paper, Llama (and therefore DiVA) produces romanized, rather than native script, responses in Chinese and Japanese which leads to the reduced performance in those languages compared to BLSP-KD. 

There are always many models which can be included as baselines, especially in quickly growing research areas. We feel our selection of SALMONN, Qwen Audio, and Qwen 2 Audio as primary baselines instead is supported by: 
1. Their state-of-the-art performance on chat-style benchmarks and paralinguistic tasks as shown in the recent Qwen 2 technical report in which these are the top 3 performing models [1].
2. Open availability of model weights and inference code which makes evaluation of them possible without re-implementing and retraining the models. 
3. Their widespread usage overall as the most commonly used reference point and/or foundation models for other Speech LLM works. This is also likely due to their open-access release which makes their usage possible in downstream research, similar to DiVA.

[1] Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., ... & Zhou, J. (2024). Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731973971212

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
On Our Use and Adaptation of the Q-Former Architecture

comment:
In the second of our empirical results in response to reviewer comments, we'd like to address the concerns on our use of the Q-Former and clarify how our work improves over other usages of Q-formers.

We agree with Reviewer Rub3 that the Q-Former is indeed well-established in this domain, as acknowledged in our paper (lines 168-170). We also acknowledge Reviewer uKgu's point about potential limitations of the Q-Former! We noted ourselves that attention-based pooling typically increases data requirements compared to simpler heuristic pooling methods (lines 173-174).

## Novel Contribution

Our contribution on the particular topic of the architecture of the adapter is simple, but empirically effective: initializing the Q-Former using the Whisper Decoder with static query tokens (discussed on lines 176-181). While this approach limits our architecture choices somewhat, we hypothesized that leveraging the pretraining of the Whisper Decoder would be more practically beneficial. To provide stronger evidence for this hypothesis, we conducted additional empirical experiments.

## Experimental Setup

We performed three comparative training runs:

1. Our proposed approach
2. Control 1: Q-Former with random initialization
3. Control 2: MLP with random initialization

All other training parameters remained constant across runs including training data order, other component initializations, and hyperparameters.

## Results and Analysis

### Performance Across Tasks

We see that much lower [training](https://ibb.co/hZC32ky) and [evaluation](https://ibb.co/sKDptSy) loss are achieved by the approach we take in our paper v.s. either of the controls. We have attached the training and evaluation loss curves in anonymous links here.

Beyond lower loss, our empirical results demonstrate significant improvements across most tasks. Most notably, the Q-Former initialization is critical to achieving non-zero translation performance:

| Task Type | Task Name | Q-Former w/ Whisper Init | Q-Former w/ Random Init | MLP w/ Random Init |
|-----------|-----------|-------------------------|------------------------|-------------------|
| Question Answering | SD-QA | **53.58%** | 45.94% | 25.73% |
| Question Answering | HeySQUAD | **55.17%** | 47.63% | 29.48% |
| Emotion Recognition | MELD | **41.34** | 39.88 | 8.74 |
| Emotion Recognition | IEMOCAPS | **50.60** | 44.08 | 19.85 |
| Sarcasm Detection | Mustard | 52.61% | 55.07% | **56.52%** |
| Humor Detection  | URFunny | 50.24% | 50.05% | 50.28% |
| Translation | CoVOST | **13.80** | 0 | 0 |

Qualitatively, the MLP connector seems to learn only to produce the first token correctly and then immediately generates EOS afterwards. This leads to reasonable classification performance, but poor performance in generative tasks. On the other hand, the randomly initialized Q-Former is able to generate coherent QA responses, but fails to adapt to varied text-based instructions leading to zero BLEU scores in translation.

### A Note on Length Generalization

This limitation applies equally to MLP with average pooling since it produces tokens at a fixed-ratio to the input tokens. The Whisper Encoder (also used in Qwen 1 & 2, SALMONN, BLSP, and BLSP-KD) produces mixed 1500 tokens for all input lengths and has a maximum input length of 30 seconds. As noted in our response to Reviewer uKgu on the 448 token limit, in practice, this limitation can be mitigated both for DiVA and other models that use the Whisper Encoder by encoding multiple chunks of 30 seconds for length generalization.

## Conclusion

We hope that these additional experimental results address the reviewer concerns by showing that:

1. The Whisper-initialized Q-Former generally improves over the suggested simpler alternative from Reviewer uKgu
2. Our contribution on this particular front is not the architecture itself, but instead a simple but previously unused initialization strategy to reduce the data demands of attention-based pooling. This adjustment is likely to benefit not just DiVA, but other approaches which use the Q-Former for SFT based training.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733279198808

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Comment of Review

comment:
We would like to thank the reviewers for their time!  We are glad to hear from the reviewers that there are limited concerns about technical soundness after the discussion period, despite remaining questions about novelty. The main open soundness concerns are from Reviewer WQdk, who unfortunately we did not have a chance to discuss further with after their initial review. We hope that similar to Reviewer uKgu "the additional experiments help address most of [their] concerns", given that they had shared concerns about the baselines compared to. 

While we understand the reviewers concern that the novelty of our work is not a "big leap", we want to highlight the following practical merits of our work:
- The reviewers agree that the model is clearly empirically effective and outperforms similarly situated models. We feel the practical utility of a strong open-weights model which can be used for interpretability studies, further refinement, and self hosted application can be significant.
- Releasing the full code to reproduce such a model on the openly-accessible infrastructure we used to train DiVA meaningfully contributes to open and reproducible science and enables future works leveraging the underlying code for scalable training.
- The ablations studying the effects of our varied contributions help practitioners understand each separately, allowing them to utilize different useful components even if they do not use all of the methods we study.

As the discussion period is coming to an end, we want to close with a final expansion on these points.

## DiVA has Strong Empirical Results For Downstream Usage:
- While individual components of our work may be seen to lack novelty, DiVA demonstrates a practical advance in speech-LLM integration. Following our additional baseline experiments, all of the reviewers who responded agreed that our model's performance is clearly stronger than other end-to-end speech models.
- Unlike some recent works the reviewers mentioned, our work releases model weights which can be finetuned further for specific tasks, styles, or languages. We contacted the authors of these works and either did not hear back or were told the weights will not be released due to corporate policy. Given these works seem unlikely to be released for further usage or study, releasing an open model with state-of-the-art capabilities is especially likely to enable new work from the broader community.
- Our open-source implementation supports reproducibility, which RRHC highlighted as a strength: "Authors use publicly available datasets and fine-tuned models". By releasing training code which uses computing infrastructure which is freely available to academics, our work could enable more practitioners to enter and advance this research topic.

## Multiple Methods, One Effective Model
- While Reviewer Rub3 noted that "using q-former with pre-trained model weights is not really in and of itself a novel technique," they acknowledged it is "definitely a noteworthy and useful method as part of a greater work". Notably, this initialization method is one part of what leads to DiVA's success and complements our separate distillation components.
- Reviewer uKgu notes that "ablation analysis confirms the distinct contributions of cross-modal token alignment and embedding distillation". This provides more nuance to understand why DiVA is so empirically effective with regards to other distillation works.
- While Reviewer Rub3 notes that other works have explored $L_2$ or feature based losses, we feel these works are meaningfully different from ours. The most notable difference is that, specific to our case of adding a new modality to an otherwise frozen LLM, we show that feature distillation leads to an equivalent solution to KL Divergence distillation at reduced cost due to the shared components.

## DiVA offers Comprehensive Evaluation and Analysis
- As demonstrated in our response tables, we conducted thorough comparisons across architectures, losses, and to all baselines which were requested and available. Across these, we have demonstrated that DiVA is effective and often exceeds the performance of models trained on far more data. We are glad to see that there are limited soundness concerns about which tasks we selected given DiVA achieves "strong results on [this] wide variety of tasks" [RRHC].
- As uKgu noted, this includes "a qualitative user study [which] shows DiVA's strong alignment with human preferences in conversational quality". This pairwise preference study for Speech LLMs is a first of its kind and we release the open-source code to enable future such studies by others. We hope that the contribution of this first pairwise preference study of Speech LLMs clearly highlights the benefits of this approach, since as we note on line 485, our results provide early indications that many current "benchmarks may not correlate with practical usage."
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces DiVA (Distilled Voice Assistant), a model for training speech-based large language models (Speech LLMs) without requiring explicit instruction-following data in the speech modality. The key components of the approach are: 1) integrating a Whisper-based speech encoder with a text model initialized from Llama weights, allowing the model to process both speech and text inputs. The text model remains frozen while the combined model is trained on a relatively small amount of data, including CommonVoice. 2) training the model using dual distillation and alignment loss techniques, with cross-modal alignment loss to align speech and text modalities and embedding distillation loss to align output representations of speech and text. With these components, DIVA demonstrates competitive performance across several task groups, including spoken question answering, speech classification, and speech-to-speech translation, and surpassing baseline methods, such as Qwen2-Audio.

Strength of this paper

- Competitive performance: DiVA surpasses previous SOTA models, like Qwen2-Audio, with significantly lower resource requirements, making it a more resource-efficient solution. Evaluations show competitive results across various tasks and benchmarks, with DiVA performing well in instruction-following and cross-modal alignment. These experiment results support the model’s efficacy.
- A cost-effective methodology of using ASR data alone to improve instruction-following behavior and reducing reliance on annotated instruction data. The paper also demonstrates that it is sufficient to teach an existing instruction-tuned model to understand audio using relatively lightweight techniques, effectively enabling the model to follow instructions via audio queries.
- The paper is clearly written, and it uses publicly available datasets and fine-tuned models, which increases the paper's accessibility and reproducibility.


Weakness of this paper

- Limited Novelty: the method lacks novelty, as many components (e.g., attention-based pooling like Q-former, L2 loss as a replacement for KLD) are well-known in existing literature. Combining these pieces to process audio input with Llama models, though interesting, has been explored in previous works like Nvidia's SpeechLLM and SALMONN, which also use similar strategies. The proposed distillation approach, which relies on self-supervised and cross-modal transfer methods, does not go beyond well-established techniques and lacks meaningful innovation, limiting its impact.
- Unclear Model Outputs and Evaluation: The paper does not clarify how speech (waveform) is generated from the model output, leaving a gap in understanding how the system produces audible speech. There is no direct evaluation of speech quality, such as MOS (Mean Opinion Score) experiments, which would be valuable to assess the quality of the generated speech. The speaker similarity and translation quality (ASR-BLEU) metrics are missing for the speech-to-speech translation task. These would help evaluate the overall quality of the translations and speaker consistency. Besides, The evaluation of the voice assistant claims is unclear. The paper mentions the creation of a voice assistant but does not explain or evaluate what this entails, raising questions about the paper's practical applications and performance in real-world tasks.
- Lack of Comprehensive Comparisons: The paper does not compare the method with important works, such as those using cross-entropy or other distillation methods (e.g., BLSP, AudioChatLlama), which would clarify the advantages and uniqueness of the proposed approach. A more detailed comparison with existing methods like Qwen2-Audio and the Llama-3 backbone is needed, as claims about DiVA outperforming state-of-the-art models may be misleading due to discrepancies in model architectures and backbones.
- Lastly, related work and references is not incomplete, e.g., omitting numerous relevant works in the Speech LLM and instruction-following domains. This makes it harder to understand the position of this work, and what's the trade-offs and how proposed work compared with existing approaches, to help readers better understand the method's limitations and future applicability.