Venue: NAACL 2025

# Enhancing Temporal Understanding in Audio Question Answering for

Large Audio Language Models

 Arvind Krishna Sridhar &Yinyi Guo &Erik Visser

Qualcomm Technologies Inc. Qualcomm Technologies Inc. San Diego, CA arvisrid@qti.qualcomm.com yinyig@qti.qualcomm.com evisser@qti.qualcomm.com

###### Abstract

The Audio Question Answering (AQA) task includes audio event classification, audio captioning, and open-ended reasoning. Recently, AQA has garnered attention due to the advent of Large Audio Language Models (LALMs). Current literature focuses on constructing LALMs by integrating audio encoders with text-only Large Language Models (LLMs) through a projection module. While LLAMs excel in general audio understanding, they are limited in temporal reasoning, which may hinder their commercial applications and on-device deployment. This paper addresses these challenges and limitations in audio temporal reasoning. First, we introduce a data augmentation technique for generating reliable audio temporal questions and answers using an LLM. Second, we perform a further fine-tuning of an existing baseline using curriculum learning strategy to specialize in temporal reasoning without compromising performance on fine-tuned tasks. We demonstrate the performance of our model using state-of-the-art LALMs on public audio benchmark datasets. Third, we implement our AQA model on-device locally and investigate its CPU inference for edge applications.

1026 Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track), pages 1026-1035 April 30, 2025 (c)2025 Association for Computational Linguistics

## 1 Introduction

Multimodal Question Answering (MQA) involves generating relevant answers for multimedia inputs such as images, audio, and video, in response to user queries Pan et al. (2024). Following the success of large pretrained transformer models for MQA, audio-specialized question answering has gained traction. Audio Question Answering (AQA) is an audio-to-text task where, given an audio file and a question, the model produces an answer by analyzing the audio content.

**Audio Question Answering:** Recent literature Gong et al. (2023); Ghosh et al. (2024a); Tang et al. (2024); Deshmukh et al. (2023) in AQA develops end-to-end pretrained transformer-based architectures known as Large Audio Language Models (LALMs). Figure 1 provides a general framework for our AQA model architecture Gong et al. (2023). It comprises three components: an audio encoder, a projection module, and a text decoder. The Audio Spectrogram Transformer (AST) Gong et al. (2021) encodes the input audio clip into spectrogram feature representations. The projection module converts these audio feature representations into texts equivalent embeddings for the text decoder. The LLaMA model serves as the text LLM decoder, taking the converted audio feature embedding and the question as input. During training, we add metadata as an optional input that is generated by the proposed data augmentation in Section 2.1. It helps provide extra guidance to the LLM decoder along with the text projections of the audio clip and aids in the overall audio-text representation learning. The GAMA model Ghosh et al. (2024a) follows a similar architecture to LTU Gong et al. (2023), combining multiple types of audio features, including activations from multiple layers of AST, Audio Q-former, and a soft prompt that provides audio events information. In this paper, we intend to discuss a few problems and limitations that we discovered in the process of developing a LLLM for commercial edge devices and explain our proposed techniques to overcome them. We chose LTU as the base model for our experiments over GAMA due to the ease of on-device implementation.

**Use Case Motivation:** Although LALMs excel at general audio understanding and have shown good overall performance in audio captioning, classification tasks, and open-ended reasoning tasks, there is a significant gap between LLM research and real-world product requirements. First, LALMs fine-tuned end-to-end with millions of audio-text samples do not capture fine-grained audio understanding well. Their performance isn't impressive on specialized reasoning tasks that require fine-grained understanding, such as temporal reasoning

(Gong et al., 2023). Audio temporal reasoning is the ability to understand the temporal context and relationship between events in the input media. Specialized audio temporal understanding has significant potential across various sectors for commercial adoption. In healthcare, it can be used for continuous monitoring and analysis of heartbeat and respiration over a period of time and provide useful analysis and recommendations to the user. In smart homes, it can enable advanced security monitoring with privacy protection by capturing and analyzing the sequence of events in live stream audio coming from sensors located in multiple areas. (Gong et al., 2023) explains that the lack of fine-grained understanding in LALMs might be due to performing temporal downsampling at the audio encoder-projection module juncture, which is a trade-off to save computational efficiency and limited training data for temporal analysis. In this paper, we address both these limitations while also keeping in mind the limitations in commercial LALMs, including low memory footprint, ease of on-device implementation, reliability, and minimal training compute. Due to the difficulty in procuring large amounts of pretraining data, expensive compute power, and time constraints, it is painstakingly difficult to retrain an LALM from scratch for improving a particular skill. On top of that, the large memory requirements of LALMs make it difficult to run them on low-compute edge devices.

**Existing Work on Temporal Reasoning in AQA:** In this paper, we focus on optimal training pipeline strategies to improve audio temporal understanding. Before the pre-trained transformers era, DAQA Fayek and Johnson (2020) and ClothoAQ Lipping et al. (2022) proposed a synthetic rule based and crowd sourced audio temporal reasoning datasets respectively. Ghosh et al. (2024) published an annotated benchmark to evaluate the audio encoders on compositional reasoning including order or occurrence of acoustic events. Yuan et al. (2024) discuss the limitations of CLAP encoder in capturing temporal information and propose a data augmentation strategy to improve the same.

**Motivation for Deploying AQA on Edge:** With the large memory requirements of LALMs scaling billions of parameters, the inference becomes expensive to run on cloud GPUs Desislavov et al. (2023). For commercial audio understanding use cases, such as smart home Internet of Things (IoT) and industrial IoT, where we can capture streams of audio from various sources such as machinery, front door, kitchen, etc., using a simple audio receiver, we need the AQA model on an always-on low-powered edge device for reasonable inference cost and preserving privacy by performing computation of audio on a self-contained edge CPU.

**Contributions:** To the best of our knowledge, we are the first to investigate the problem and limitations of audio temporal understanding in LALMs and address them from a commercialization perspective. Our contributions in this paper are as follows: First, we propose a data augmentation technique to reliably generate audio temporal question and answer pairs using GPT-4. Second, we show that fine-tuning the baseline checkpoint via curriculum learning helps improve the model's temporal awareness and reasoning without losing its original performance. Finally, we implement the AQA to run on CPU locally for commercial edge applications.

## 2 Methodology

We divide our proposed methodology into two sections. First, we explain the data augmentation strategy for generating temporal reasoning data. Second, we discuss our temporal fine tuning strategy.

### Temporal Reasoning Data Augmentation

In order to improve the temporal reasoning capability of generalist LALMs, we developed a data augmentation technique that ensures the questions are intuitive to human temporal perception. We prompt GPT-4 OpenAI et al. (2024) with the audio clip's ground truth, such as audio event labels, audio captions, and their metadata comprising timestamps of audio events. For Audioset-SL, we use audio clips containing fewer than seven types of foreground sound events with a total occurrence number of less than ten and consider sounds with a duration longer than nine seconds as background sound. We state the temporal information of the sound events in natural language and use that as part of the prompt for GPT-4 to generate question-answer pairs with rationale. In the prompt, we include few-shot paired examples of temporal reasoning tasks, including temporal relationships, chronological ordering, duration comparison, and occurrence counting. Table 1 showcases the different types of metadata-question-answer pairs generated by our temporal data augmentation technique. We refer to the data generated by this pipeline as TemporalQA. We preprocessed the question-answer pairs to filter 

* [105] M. C. Bock, J. A. Krizhevsky, and D. R. Hinton. Learning to learn a sequence from the data set. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-246, 2009.
* [106] S. Ghemawatwat, E. Hinton, and I. Goodfellow. Imagenet classification with deep neural networks. In _Advances in Neural Information Processing Systems (NIPS'08)_, pages 1398-1405, 2008.
* [107] T. Hariharan, P. Dollar, and R. Girshick. Generative adversarial nets for image recognition. In _International conference on computer vision and pattern recognition_, volume 1, pages 103-113, 2008.
* [108] Y. Bengio, L. Bottou, and X. Bengio. Generative adversarial nets: A survey. _Journal of Machine learning research_, 2015.

[MISSING_PAGE_POST]

 

out wrongly formatted data. To assess the quality of GPT-4 generated question-answer pairs, we conducted a human evaluation to score on correctness, reasoning quality and hallucination as shown in Table 2. Correctness measure checks if the answer is correct for the given question. Reasoning quality ensures that the accompanying reason is meaningful and helps in arriving at the answer. Hallucination refers to audio events/phrases present in the answer that are not mentioned in the metadata. In the Table 2, the near perfect scores for correctness and reasoning quality and low hallucination rate of the generated question-answer pairs reflects the high quality of generated temporal reasoning data.

### Temporal Finetuning via Curriculum Learning

In this section, we outline the training strategy employed to integrate temporal reasoning capabilities into a Large Audio Language Model (LALM) designed and finetuned for general audio understanding. To learn temporal reasoning skill on an already finetuned AQA model, we adopt a curriculum learning approach that merges TemporalQA with a few core finetuned Audio Question Answering (AQA) tasks, including audio classification and audio captioning. We conducted an empirical investigation to determine the optimal types of AQA tasks and the appropriate ratio of new skills (temporal reasoning) to existing skills. Based on our analysis and hyperparameter tuning, we observed that a 50:50 ratio of temporal reasoning to core AQA tasks--comprising audio event tagging, audio label classification, and audio captioning--combined with a learning rate ten times lower than that of the original finetuning, is optimal for learning temporal reasoning skills without significantly compromising the model's original performance. We refer to our temporal finetuned model with and without metadata on LTU base as AQA+Temp-M and AQA+Temp, respectively.

\[T_{\text{total}}=T_{\text{temporal}}+T_{\text{core AQA}}, \tag{1}\]

Where T refers to training data and the + operation combines both operand datasets with a random shuffle. We also provide metadata of audio, such as audio events and background noise information, in natural language in the text prompt as guidance to mitigate the information bottleneck at the projection module.

## 3 Experiments

### Datasets

We choose the LTU model Gong et al. (2023) as our baseline. We adopt a similar training dataset accruing strategy to Gong et al. (2023). Our initial stages of curriculum learning focus on training the audio encoder and projection model with a combination of audio event classification public datasets, including Audioset, FSD50k, VGGSound, and Feresound, and audio captioning public datasets, such as Clotho and Audiocaps Gong et al. (2023). We use Audioset-strong labelled Hershey et al. (2021) and FSD50k datasets to synthetically generate 20k temporal reasoning data using the data augmentation strategy explained in Sec 2.1. TemporalQA has an 80:20 train-test split. We adopt the inference style of Gong et al. (2023), including the generation of audio descriptions for the FSD dataset. All audio clips are truncated to 10s to fit the audio encoder context window.

### Experiment Setup

We train the AQA architecture from scratch with four-stage curriculum learning as described in Gong et al. (2023). For temporal reasoning fine-tuning, we perform model parallelism-based distributed training on 8 A100 GPUs for 2 epochs with a learning rate of 1e-4 and cross-entropy as the loss function. We found that a batch size of 24 and a micro-batch size of 1 work best for specializing the model further on a single task, as opposed to a batch size of 256 and a micro-batch size of 16 for fine-tuning from scratch. The low-rank adaptors (LoRA) hyperparameters alpha and r are set to 16 and 8, respectively. We set \(\alpha_{1}\) and \(\alpha_{2}\) to 1 while keeping \(\alpha_{3}\) at 0 to provide equal weightage for answer and reason and for a fair comparison with conventional metrics.

### On-device Implementation

To run the AQA model on CPU, we perform 16-bit and 8-bit post-training quantization as mentioned

\begin{table}
\begin{tabular}{|l|l|} \hline
**Metrics** & **Score** \\ \hline Correctness & 4.98 \\ \hline Reasoning Quality & 4.99 \\ \hline Hallucination & 0.02 \\ \hline \end{tabular}
\end{table}
Table 2: Human evaluation of the GPT-4 generated question answer pairs. All the metrics score range from 0 to 5. For correctness and reasoning quality, higher score is preferred while for hallucination, a lower score is optimal.

 

in llama.cpp. We implement the AQA architecture on top of the C++ implementation of LLaMA in the llama.cpp framework. First, we merge the LoRA weights into the LLaMA model of AQA+Temp and convert the checkpoint to gguf format. Second, we implement the audio encoder and projection module in onnxruntime to combine their outputs with the LLaMA in C++. We perform the experiment to measure inference speed on 100 randomly sampled questions from our test set of AQA described in 3.1 and report the average.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|} \hline Model & \multicolumn{2}{c|}{Clotho} & \multicolumn{2}{c|}{AudioCaps} & \multicolumn{2}{c|}{FSD} & \multicolumn{2}{c|}{TemporalQA} \\ \cline{2-9}  & SPIDER & FENSE & SPIDER & FENSE & SPIDER & FENSE \\ \hline LTU & 0.19 & 0.56 & 0.31 & 0.67 & **0.08** & **0.47** & 0.27 & 0.57 \\ \hline GAMA & 0.04 & 0.41 & 0.09 & 0.55 & 0.05 & 0.42 & 0.22 & 0.65 \\ \hline \hline AQA+Temp(w/o CL) & 0.04 & 0.46 & 0.04 & 0.49 & 0.06 & 0.33 & 0.52 & 0.64 \\ \hline AQA+Temp-M(w/o CL) & 0.04 & 0.36 & 0.04 & 0.48 & 0.07 & 0.32 & 0.43 & 0.63 \\ \hline \hline AQA+Temp (Ours) & 0.24 & 0.61 & 0.38 & 0.71 & 0.06 & 0.44 & 0.48 & 0.66 \\ \hline AQA+Temp-M (Ours) & **0.31** & **0.62** & **0.43** & **0.73** & 0.07 & 0.43 & **0.70** & **0.73** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of performance on LTU baseline with proposed finetuning on temporal reasoning. Temp refers to temporal finetuning and Temp-M refers to temporal finetuning with meta data information. w/o CL refers to training AQA on temporal reasoning data without curriculum learning.

Figure 3: Barplot of LTU and GAMA baseline and temporal finetuned checkpoints for temporal dataset.

 



## 4 Results

### Quantitative Analysis of Temporal Finetuning

Table 4 shows the performance of the proposed temporal fine-tuning for temporal reasoning with LTU as the base model. For a fair evaluation, during inference, we do not provide metadata to the models. After temporal fine-tuning, there is a considerable increase in all the metrics across datasets except for FSD. This might be due to differences in the format, adopted from LTU Gong et al. (2023), of FSD dataset's groundtruth and LALM's response. FSD is an audio classification dataset while the other datasets in evaluation are natural language description based datasets. FSD has a list of audio events as label while the LLMs generate an audio caption style answer. For example, the ground truth FSD label is "Electric guitar; Guitar; Plucked string instrument; Musical instrument; Music" while the generated audio caption is "Music is playing with a plucked string instrument and a bass guitar, creating a rich and dynamic soundscape.". For a reliable accuracy, in future, we can convert the audio event labels of FSD into natural language sentence using an off-the-shelf LLM and train our LALM on uniform response format. The significant improvement of Spider and FENSE metrics for AQA+Temp-M over LTU shows that we can offset the information bottleneck at the projection layer to some extent with extra textual guidance. It is notable that our AQA+Temp and AQA+Temp-M models performs better than the GAMA baseline, which has a sophisticated audio encoding. This emphasizes the need for good data augmentation in addition to architectural improvements. From the reasonable improvement in scores across all the datasets of AQA+Temp-M compared to AQA+Temp, we infer that providing metadata during training helps in better detection of audio events and improved audio-text representation mapping. In Fig 3, our proposed models show consistent improvements over the baseline, indicating the effectiveness of temporal finetuning. Table 5 presents the performance of various models on the MMAU Test-Mini Sound split benchmark Sakshi et al. (2024). Based on our organization's guidelines, we use the test-mini instead of the full test set as the latter requires us to upload our model's generations to the MMAu webpage. Our proposed method, AQA+Temp-M, performs better than the baseline LTU by a significant margin of 10.21. This shows the efficacy of our proposed data augmentation and temporal finetuning. Hence, the proposed method improves temporal reasoning in the baseline LALM while maintaining previously learned skills, as illustrated quantitatively in Table 4 and 5.

### Qualitative Analysis of Temporal Finetuning

From Table 3, it is evident that temporal finetuning with temporal reasoning data augmentation, as described in Section 2.2, results in the generation of rationale with temporal commonsense knowledge compared to the baseline. In the first example, although the baseline's answer is correct, the reasoning is wrong since the model is only provided with 10 seconds of audio clip content. In the second example, the baseline model states incorrect audio events--tap and generic impact sounds--and continues to use them in the rationale. On the other hand, the AQA+Temp generates the correct temporal answer along with a plausible explanation as rationale. This illustrates a qualitative improvement in our proposed method's answer generation over the baseline.

### Ablation Study on Meta data and Curriculum Learning

We conduct an ablation study on the design choices, namely, providing meta data information and learn

\begin{table}
\begin{tabular}{c c c} \hline \hline Model Name & Size & Accuracy(\%) \\ \hline Random Guess & - & 26.72 \\ Most Frequent Choice & - & 27.02 \\ Human (test-mini) & - & 86.31 \\ \hline Pengi & 323 M & 6.1 \\ Audio Flaming Chat & 2.2B & 23.42 \\ \hline M2UGen & 7B & 3.6 \\ LTU & 7B & 22.52 \\ LTU AS & 7B & 23.35 \\ MusiLingo & 7B & 23.12 \\ MuLLaMA & 7B & 40.84 \\ GAMA & 7B & 41.44 \\ GAMA-IT & 7B & **43.24** \\ \hline Qwen-Audio-Chat & 8.4B & 55.25 \\ Qwen2-Audio & 8.4B & 7.5 \\ Qwenz-Audio-Instruct & 8.4B & 54.95 \\ SALAMONN & 13B & 41 \\ Gemini Pro V.1.5 & - & 56.75 \\ GPT4o + weak cap. & - & 39.33 \\ GPT40 + strong cap. & - & 57.35 \\ LLama-3-Instruct + weak cap. & 8B & 34.23 \\ LLama-3-Instruct + strong cap. & 8B & 50.75 \\ \hline AQA+Temp (Ours) & 7B & 28.83 \\ AQA+Temp-M (Ours) & 7B & **32.73** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results on MMAU Test-Mini Sound Split

ing with curriculum learning. In Table 4, the LTU model shows the baseline performance. The second section comprising of AQA+Temp (w/o CL) and AQA+Temp-M (w/o CL) reflects our model's performance without curriculum learning while the last two rows, AQA+Temp (ours) and AQA+Temp-M (ours) uses curriculum learning. Without curriculum learning, the AQA+Temp and AQA+Temp-M models perform poorly on all the datasets except TemporalQA. This is expected as the model forgets it's base checkpoint finetuning and overfits to temporal reasoning. Another interesting observation is that AQA+Temp-M performs better than AQA+Temp only when trained with curriculum learning. This could be due to better learning of the audio-text embedding due to a combination of multiple audio tasks - audio tagging, audio captioning and audio question answering. This analysis emphasizes the joint importance of curriculum learning and meta data information.

### Insight on On-device AQA Inference

Table 6 presents the model loading time and inference speed of AQA for different floating point precisions. The load time denotes the time taken to load the model into the CPU. Prompt Eval Rate measures the number of user query prompt tokens encoded relative to the time taken for performing audio and prompt encoding. Eval rate refers to the time taken to generate the response. User prompts should usually be encoded quicker than the response generation because user prompts can be encoded as a batch of tokens while a response is generated auto-regressively, word by word. Despite this, for the 4-bit and 16-bit models, we see a lower Prompt Eval Rate than Eval Rate. This could be due to the audio encoding overhead, which needs to be kept in mind for improving overall inference latency.

## 5 Conclusion

In this work, we proposed a novel data augmentation strategy to generate temporal reasoning QA pairs using LLMs. Next, we finetuned a SOTA AQA model on the generated temporal reasoning data and showcased quantitative improvements across evaluation metrics. Finally, we showcased our implementation of the AQA model on-device and studied its performance. In the future, we will reduce the memory footprint of our AQA model to be able to fit into low-powered devices. This will also significantly reduce the active RAM usage and boost encoding and decoding speeds. Also, we plan to investigate quantization-aware fine-tuning techniques and study the generation quality vs. quantization trade off. We plan to introduce an evaluation metric that can appropriately select the facts from the answer and compare them against the ground truth. We can use the metric as a loss term during fine-tuning of the AQA model to prioritize the learning of specialized skills reliably.

## References

* Deshmukh et al. (2023) Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An audio language model for audio tasks. In _Advances in Neural Information Processing Systems_, volume 36, pages 18090-18108. Curran Associates, Inc.
* Dessislavov et al. (2023) Radosvet Dessislavov, Fernando Martinez-Plumed, and Jose Hernandez-Orallo. 2023. Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning. _Sustainable Computing: Informatics and Systems_, 38:100857.
* Fayek and Johnson (2020) Haytham M. Fayek and Justin Johnson. 2020. Temporal reasoning via audio question answering. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 28:2283-2294.
* Ghosh et al. (2024a) Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024a. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. _Preprint_, arXiv:2406.11768.
* Ghosh et al. (2024b) Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, Ramaneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024b. Compa: Addressing the

\begin{table}
\begin{tabular}{c c c c c} \hline \hline FP (bits) & Model Size (GiB) & Load Time (ms) & Prompt Eval Rate (TPS) & Eval Rate (TPS) \\ \hline
16 & 12.55 & 10925.84 & 6.95 & 7.35 \\
8 & 6.67 & 2690.79 & 13.57 & 13.16 \\
4 & 3.56 & 1395.71 & 15.79 & 19.64 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of inference speed for AQA across different floating point (FP) precision on-device. FP and TPS refers to floating point precision and tokens per second respectively.

 

gap in compositional reasoning in audio-language models. In _The Twelfth International Conference on Learning Representations_.
* Gong et al. (2021) Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. In _Proc. Interspeech 2021_, pages 571-575.
* Gong et al. (2023) Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. 2023. Listen, think, and understand. _arXiv preprint arXiv:2305.10790_.
* 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 366-370.
* Lipping et al. (2022) Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. 2022. Clotho-aqa dataset.
* OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, and Lama Ahmad et al. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Pan et al. (2024) Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. 2024. Chain-of-action: Faithful and multimodal question answering through large language models. _arXiv preprint arXiv:2403.17359_.
* Sakshi et al. (2024) S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. Mmau: A massive multi-task audio understanding and reasoning benchmark. _Preprint_, arXiv:2410.19168.
* Tang et al. (2024) Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. SALMONN: Towards generic hearing abilities for large language models. In _The Twelfth International Conference on Learning Representations_.
* Yuan et al. (2024) Yi Yuan, Zhuo Chen, Xubo Liu, Haohe Liu, Xuenan Xu, Dongya Jia, Yuanze Chen, Mark D. Plumbley, and Wenwu Wang. 2024. T-clap: Temporal-enhanced contrastive language-audio pretraining. _Preprint_, arXiv:2404.17806.

## Appendix A OnDevice Graphical User Interface Examples

Figure 4 and 5 shows the GUI and an example sample for AQA running on edge CPU.

## Appendix B System Prompts

The System Prompts used for generating temporal question answering data and for on-device inference are shown in Table 7.

## Appendix C Sample Conversation with AQA

Figure 6 shows a sample conversation with AQA on an audio file recorded in an industrial setting.

## Appendix D Device Specifications for the on-device demo

The Device has an ARM-based Snapdragon(R) X Elite processor with 32.0 GB RAM (31.6 GB usable). The CPU has 3.42 GHz clock speed operating on a 64-bit operating system.

## Appendix E Additional Annotation Details

For the human evaluation to assess the quality of GPT-4 generated question answer pairs, we recruited 2 annotators through advertisement inside the department. We randomly sampled 100 metadata-question-answer pairs and provided to the consented annotators in the form of a double blindfolded survey. Therefore, not required by our IRB to be reviewed by them. The authors of this work are not lawyers. However, this opinion is based on the United States Federal regulation 45 CFR 46, under which this study qualifies for exemption via 46.104 exempt research 



## 6 Conclusion

\begin{table}
\begin{tabular}{|p{284.5pt}|p{390.1pt}|} \hline Stage & System Prompt \\ \hline Temporal Data Generation & Generate 5 questions and answer pairs along with metadata from the following information about the audio. The questions are used for temporal audio question answering task. Assume the audio description and audio event time information as the audio file itself. Do not ask questions whose answers are not present in the description. Write the answers in a more explanatory and human friendly manner. You can add some common senses or facts whenever it is possible along with the answer. Format each question in a single line as a JSON dictionary with keys – “id”, “question”, “answer”, ”metadata”. Some examples of questions you could ask are : What sound events occurs first? What sound comes after the male speech at the beginning? (if male speech is present in the description) What event happens before the engine running sound? Which event occurs towards the end ? Is the door bell sound after the dog barking? Answer true or false and provide your reasoning steps. Can you hear footsteps before the baby cries? Answer true or false and provide your reasoning steps. What is the chronological order of the sound events? What is the background sound if there’s any? Please generate diverse questions with paraphrasing. \\ \hline AQA On-device Inference & A chat between a curious user and an audio question answering artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions. You are given an audio clip and a question from the user. Do not generate false audio events or hallucinations that are not there in the audio clip. Do not contradict yourself without proper evidence. \\ \hline  & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 7: Systemp Prompts for Temporal Data Generation and AQA On-device Inference.

Figure 4: I: Graphical User Interface (GUI) of Audio Question Answering on ARM CPU.





**Context:** Audio file comprises a 10 second recording from an industry manufacturing area. Vehicle sounds are heard initially and later during the middle part of the audio. It is followed by speech, explosion and fire alarm.

**Conversation**

User: Describe the audio in detail.

AQA: The audio features a vehicle starting up, followed by speech, another instance of the vehicle sound, an explosion, more vehicle sounds, and finally a fire alarm.

User: Where do you think this audio was recorded?

AQA: Based on the audio content, it seems like the audio was recorded at a construction site where vehicles are being used for transportation and heavy machinery is also present.

User: When do you hear people talking and fire alarm sound?

AQA: The people start speaking at around 0.5 seconds into the audio, while the fire alarm sound can be heard at around 3.2 seconds.

User: Do you hear any babies?

AQA: No, I don't hear any babies in the audio. It primarily focuses on the sounds of vehicles and machinery with occasional speech and alarms.

User: Guess the emotion of the scene.

AQA: The scene likely has a sense of urgency due to the presence of the explosions and fire alarm, which could indicate danger or emergency situations. However, the presence of people speaking, and vehicles suggests some level of activity and productivity despite these challenging circumstances.

Figure 5: II: Graphical User Interface (GUI) of Audio Question Answering on ARM CPU.

Figure 6: Example of a conversation with AQA-Temp proposed model