WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables
Authors: Zhaojiang Lin
YONG XU
Kai Sun
Jing Zheng
Yin Huang
Surya Teja Appini
Krish Narang
Renjie Tao
Ishan Kapil Jain
Siddhant Arora
Ruizhi Li
Yiteng Huang
Kaushik Patnaik
Wenfang Xu
Suwon Shon
Yue Liu
Ahmed A Aly
Anuj Kumar
Florian Metze
Xin Luna Dong
Venue: ICLR 2026 Poster (2026)
Keywords: Speech Large Language Models
SLLM
Voice Assistant
Benchmark
Primary area: datasets and benchmarks
TL;DR: We present WearVox, the first benchmark designed to rigorously evaluate voice assistants in realistic wearable scenarios

Abstract: Wearable devices such as AI glasses are transforming voice assistants into always-available, hands-free collaborators that integrate seamlessly with daily life, but they also introduce challenges like egocentric audio affected by motion and noise, rapid micro-interactions, and the need to distinguish device-directed speech from background conversations. Existing benchmarks largely overlook these complexities, focusing instead on clean or generic conversational audio. To bridge this gap, we present WearVox, the first benchmark designed to rigorously evaluate voice assistants in realistic wearable scenarios. WearVox comprises 3,842 multi-channel, egocentric audio recordings collected via AI glasses across five diverse tasks including Search-Grounded QA, Closed-Book QA, Side-Talk Rejection, Tool Calling, and Speech Translation, spanning a wide range of indoor and outdoor environments and acoustic conditions. Each recording is accompanied by rich metadata, enabling nuanced analysis of model performance under real-world constraints. We benchmark leading proprietary and open-source speech Large Language Models (SLLMs) and find that most real-time SLLMs achieve accuracies on WearVox ranging from 29\% to 59\%, with substantial performance degradation on noisy outdoor audio, underscoring the difficulty and realism of the benchmark. Additionally, we conduct a case study with two new SLLMs that perform inference with single-channel and multi-channel audio, demonstrating that multi-channel audio inputs significantly enhance model robustness to environmental noise and improve discrimination between device-directed and background speech. Our results highlight the critical importance of spatial audio cues for context-aware voice assistants and establish WearVox as a comprehensive testbed for advancing wearable voice AI research.

Published as a conference paper at ICLR 2026
WearVox: AN EGOCENTRIC MULTICHANNEL VOICE ASSISTANT BENCHMARK FOR WEARABLES
Zhaojiang Lin' Yong Xu' Kai Sun Jing Zheng', Yin Huang', Surya Teja Appini', Krish Narang'. Renjie Tao Ishan Kapil Jain', Siddhant Arora 1.2f, Ruizhi Li", Yiteng Huang', Kaushik Patnaik', Wenfang Xu Suwon Shon', Yue Liu', Ahmed A Aly', Anuj Kumar Florian Metze Xin Luna Dong"
Meta, *Carnegie Mellon University zhaoji ianoojian@@mtaa com, sunkai on @meta com, yongxuêmeta... com, Junadongemeta.com
ABSTRACT
Wearable devices such as AI glasses are transforming voice assistants into always- available, hands-free collaborators that integrate seamlessly with daily life, but they also introduce challenges like egocentric audio affected by motion and noise, rapid nicro-interacctins, and the need distinguish device-directed speech from background conversations. Existing benchmarks largely overlook these complexi- ties, focusing instead on clean or generic conversational audio. To bridge this gap, we present WearVox, the first benchmark designed to rigorously evaluate voice as- sistants realistic wearable scenarios. Wear Vox comprises 3,842 multi-channel, egocentric audio recordings collected via AI glasses across five diverse tasks in- cluding Search-Grounded QA, Closed-Bood Book QA, Side-Talk Rejection, Tool Call- ing, and Speech Translation, spanning wide range of indoor and outdoor environ- ments and acoustic conditions. Each recording is accompanied by rich metadata, enabling nuanced analysis of model performance under real-world constraints. We benchmark leading proprietary and open-source speech Large Language Models (SLLMs) and find that most real-time SLLMs achieve accuracies on WearVox ranging from 29% to 59%, with substantial performance degradation on noisy outdoor audio, underscoring the difficulty and realism of the benchmark. Addi- tionally, we conduct case study with two new SLLMs that perform inference with single-channel and multi-channel audio, demonstrating that multi-channel audio inputs significantly enhance model robustness to environmental noise and improve discrimination between divice-directe and background speech. Our re- sults highlight the critical importance of spatial audio cues for context-aware voice assistants and establish WearVox as a comprehensive testbed for advancing wear- able voice Al research !
INTRODUCTION
Wearable devices, such Al glasses, are transforming voice assistants from handheld tools into always-available, body-worn collaborators. Unlike phones and smart speakers where interactions are episodic, hands-free free only by choice, and typically occur in acoustically stable environments, wear- ables operate at the edge of our attention, seamlessly integrating with daily activities like walking, commuting, and socializing. While this convenience unlocks unique challenges: egocentric audio affected by motion and wind noise, rapid micro-interactions constrained by strict latency requirements, and the need distinguish device-directed requests from side speech and background noise. Yet, existing benchmarks such as VoiceBench (Chen et al.) 2024). Spoken-CoQA (You et al., 2022). and Spoken-SQual (Lee al. 2018) focus primarily on clean audio or generic conversational scenarios, overlooking the specific complexities inherent to wearable interactions.
possibilities, it also introduces
"Jiitt first author
Work done at Meta
Vox is availahle http://///httit//////////errerer

---

Published as a conference paper at ICLR 2026
Q Search grounded
What's the of the Warniars game?
Quali alcuni posti popolari per questa cittä?
Warnior gome score..
QA
Warniors 110, Lakens 99, ten minutes left
There is famous street in the city
center that is always bustling.
Questo sembra
può dire più sulla che tipo atmosfera ha?
Where can watch Warriors game person?
Closedbook
Chase Center in Francisco, California
QA
Bidirectional speech translation
Look up my vet oppointment.
What some populor places to go out at night in this city?
Tool
calling
( nomee" calendor_lookup. pproms' (..3)
C'è una strada famosa centro
Oh wait, yes! will be right there!
della città che è sempre onimata. That sounds interesting, Con you tell me more about the street and what kind of atmosphere it has?
Side talk rejection
Hurry up! We are late!
(Silence..)
Figure 1: Examples of tasks from the WearVox dataset. The audio queries are recorded with Al glasses (transcribed in blue). The ground truth for each task is provided in text format.
To bridge this critical gap, introduce VearVez, the first wearable-speciffe voice assistant bench- mark designed to rigorously evaluate state-of sate--o--thearn art speech Large Language Models (SLLMs) in re- alistic wearable scenarios. WearVox comprises comprehensive collection of 3,842 multi-channel, egocentric audio recordings, spanning different tasks that reflect practical situations encountered by users of wearable devices such as AI glasses, both indoors and outdoors. WearVox is distin- guished by several valuable features:
1. Ego-centric, multi-channel audio: All recordings in Wear are captured from first- person perspective using Al glasses equipped with multiple microphones, simulating the audio input typical of wearable devices. The dataset is designed to reflect the complexity of real-world interactions, featuring a variety of speaker roles including the primary glasses wearer, conversational partners, and bystanders who positioned at different angles and dis- tances. This setup enables the modeling of realistic conversational dynamics, such as direct queries, interruptions, side-talk, and non-assistant-diredd speech, which are essential for robust and context-aware voice assistant performance.
2. Comprehensive environmental and acoustic coverage: WearVox covers a wide range of indoor and outdoor environments, including office spaces, cafés, cars, as well as streets, parks, and construction zones. Approximately 31 % the dialogues were recorded indoors, while 63% took place outdoors. Recordings were conducted under both quiet and noisy conditions, with 58% of the data collected in noisy environments and 42% quiet settings. The dataset features 13 different noise types, such rustling leaves and construction noise, carefully selected to represent real-world scenarios. Each audio sample accompanied by detailed metadata describing participant positions, distances, and environmental context, ensuring that the dataset captures the nuanced challenges inherent to wearable audio.
3. Diverse and realistic wearable assistant tasks: The benchmark encompasses a broad array of wearable assistant tasks, including Search-Grounded Question Answering (QA), Closed-Boof QA, Side-Talk Rejection, Tool Calling, and Speech Translation. The dataset is meticulously curated to reflect the functionalities expected of next- gexeerennotionn wearable assistants, ensuring that models are evaluated across a wide spectrum of practical and chal- lenging scenarios.
Building on the WearVox benchmark, conduct comprehensive experiments to evaluate the per- formance of state-of of-te--o-theaar open-source and proprietary SLLMS in realistic wearable scenarios. Our evaluation includes leading models such as GPT-4o (Hurst et al. 2024) and Gemini 2.5-flash (Co- manici et al. 2025). as well as open-source models like Qwen-2.5 omni (Xu et al. 2025), Gemma 3n (Team 2025a), Phi-4 multimodal (Abouelenin et al. 2025). and Kimi-Audio (Ding al. 2025
2

---

Published as a conference paper at ICLR 2026
For robust comparison, we also include two-stage pipeline combining Whisper ASR (Radford iet al., 2023) with a text LLMs GPT-5 (OpenAl 2025). This diverse set of models allows us to systematically analyze the strengths and limitations of current SLLMs when faced with the unique challenges posed by egocentric, multi-channel wearable audio. The experimental results show that most real-time SLLMs achieve accuracies Wear Vox ranging from 29% 59%, with substan- tial performance degradation on noisy outdoor audio, underscoring the difficulty and realism of the benchmark.
To assess the impact of multi-channel audio signal in wearable scenarios, we conduct a case study with two new SLLMs: one utilizing single-channel audio and another leveraging a multi--hannel approach built on the Llama 4 Scout (Team) (025b) architecture. Our findings reveal that incor- porating multi-channel audio inputs greatly improves model resilience to environmental noise and enhances the ability to differentiate between device-directed speech, side conversations, and back- ground noise. Specifically, side talk rejection accuracy increased from 85.6% to 93.9%, while overall accuracy improved from 61.9% to 66.4%. These results highlight the critical importance of spatial audio cues for enabling context-aware voice assistants in wearable applications.
2 RELATED WORK
Voice Assistant Benchmarks. The trajectory of voice assistant benchmarking has evolved from
-eReeeeedd comprehension tasks to comprehensive, end-to-end evaluation of speech LLMs.
Early datasets such as Spoken-SQuAD (Lee et al.] 2018) and Spoken-CoQA (You et al. 2022)
extended SQuAD and CoQA into the spoken domain by generating speech with TTS. Subse-
quent benchmarks, including HeySQuAD (Wu et al. (023) and LibriSQA (Zhao et al. 2024),
improved realism by incorporating large-scale houman-spoken recordings. reducing reliance on syn- thetic speech. More recent Saattts-AhdiBhehhhh (Wang et al. 2025), MMAU (Sakshi et al. 2024) VoiceBench (Chen et al., 2024). and CAVA (Held et al., (202) broaden the scope beyond QA to include speech instruction following, paralinguistic understanding, acoustic scene perception, and even music reasoning, reflecting the expanding capabilities of Audio-LLMs. Finally, the latest benchmarks such as CAVA (Held et al. (025) and FDX-bench (Lin et al. (025) emphasize real-time conversational aspects, explicitly evaluating turn-taking, latency, and duplex interaction crucial properties for practical voice assistants. In contrast, WearVox is the first voice benchmark designed specifically for wearable computing, leveraging egocentric multi-channel audio and a diverse range of real-world environments capture conversational dynamics (such side talk and non-assistant- directed speech) that are critical for advancing next-generation voice assistants. A more detailed comparison between WearVox and existing voice assistant benchmarks is provided in Table [
Speech LLMs. Modern speech LLMs push beyond ASR-LLM-TTS pipelines toward end-to-end, streaming "omni" assistants. GPT-4o (Hurst al.] 2024) integrates audio understanding and gen- eration with real-time performance, catalyzing speech-first UX in production systems. Gemini 2.5 Flash (Comanici et al. 2023) adds native audio support and controllable "thinking" budgets, balancing latency and reasoning for live dialog. In the weighweights space, Qwen2.5-Omnn (Xu jet al.] 2025). Kimi-Audio (Ding et al.) (0025). GLM4- Voice (Zeng et al.) (2024) and Phi-4 Mul- timodal (Abouelenin et al., 2022)) provide unified speech-text modeling with competitive audio- reasoning performance. Gemma 3n (Team] 2025a) extends this trend to mobile-scale audio under- standing. Full-duplex LLMs such as Moshi (Défossez et al.) (2024). SALMONN-
(Yu et al. 2024]. and SyncLLM (Veluri et al. 2024) enable listen and speak at the same time without strict turn-taking, As complement, we introduce a multichannel SLLM and share findings on the role of spatial audio, offering insights beyond the single-channel focus of existing speech LLMs.
3 WearVox DATASET
3.1 PROBLEM DEFINITION
The WearVes dataset is developed to benchmark and advance the capabilities of wearable voice assistants in real-world, egocentric audio environments. It provides comprehensive suite of tasks (as illustrated in Figure D]. diverse speaker roles, and varied acoustic conditions to facilitate robust evaluation and development of next-generation voice assistant systems.
3

---

Published as a conference paper at ICLR 2026
Egocentric Audio
Multi-channel Audio
Conversational Dynamics
Domain Diversity
Dataset Size
Audio
Benchmark
Source
Spoken-SQuAD
42K
TTS
(poken-CoQA
40K
TTS
HcySQuAD
X
173K
TTShecording
LibriSQA
X
214K
LibriSpeech
AudioBenchh
5.5K
LibriSpeech/CCie
MMAU
X
10K
Diverse public audio
VoiceBench
X
X partial
5.8K
TTShecording
CAVA
X
6K
STOP
FDX -Bench
X
< <IK
Candor/TTS
Wear Vox
3.8K
Consumer wearables
Table 1: Comparison of WearVox to existing voice assistant benchmarks.
3.1.1
TASKS FORMULATION
Wear Vox encompasses five core tasks that reflect both common and challenging scenarios for wear- able voice assistants. All tasks are formulated as a Text In, Speech In, Text Out problem, defined as:
f(T,, Sy) To
where Tj is the input text prompt, S₁ is the input speech signal, and To is the output text. Each task has distinct fine-gra mra-griieed composition, as detailed below:
1. Search-Grounded QA. Many daily queries to wearable assistants (e.g., financial news, sports scores) require up-to-date, external information. In this task, the assistant must pro- vide factual answers based on search results.
Ty: Task description and external search results
Sy: Wearer request in speech
To: Answer text
2. Closed-Book QA. The assistant responds general knowledge questions without access to external resources, relying solely on its internal knowledge.
Ty: Task description
Sy: Wearer request in speech
To: Answer in text
3. Tool Calling, The assistant is required to invoke specific tools APIs (e.g., music player, reminders) based on wearer requests.
Ty: Task description and tool/function definitions (including tool name, tool descrip- tion, and parameters)
Sy: Wearer request in speech
To: Tool call in JSON format
4. Side Talk Rejection. The system must accurately distinguish and ignore non-device-
directed speech, such as background conversations and bystander chatter.
Ty: Task description
Sy: Side talk speech, triggered by either the wearer, conversational partner, or by- standers
To: Special control token (e.g., [Mute]) suppress downstream components (such as TTS)
5. Bidirectional Speech Translation. The assistant facilitates translation between the wearer and a conversational partner who speak different languages. In this task, the assistant must perform both speaker diarization and speech translation simultaneously. We focus on of- fline, whole-dialog translation rather than simultaneous translation, simplifying the evalu- ation protocol under the assumption that performance in both settings is highly correlated.
Ty: Task description
Sy: Bilingual, multi-turn dialogue between the wearer and conversational partner
To: Diarized, translated dialogue in text
4

---

Published as a conference paper at ICLR 2026
3.1.2 SPEAKER ROLES
WearVox simulates realistic, multi-party interactions by involving three distinct speaker roles:
Wearer:
The primary user of the wearable device, who initiates most revice-directed queries and commands. In Bearch-Groundee QA, Closed- Book QA, and Tool Calling tasks, the wearer is typically the source of the spoken input (Sr). issuing questions or requests to the assistant. In Side Talk Rejection, the wearer may also produce non-device-directed speech, testing the assistant's ability to distinguish between intentional and incidental in- put. For Bidirectional Speech Translation, the wearer participates one of the two parties in the bilingual conversation, requiring the assistant to correctly identify and translate their utterances.
Conversational Partner: An individual actively engaged in dialogue with the wearer. This role is especially prominent the Bidirectional Speech Translation task, where the con- versational partner speaks different language and participates in multi-turn exchanges with the wearer. The assistant must perform speaker diarization to attribute each utterance correctly and provide accurate translations for both parties.
Bystander: A third-party speaker who may contribute incidental or background speech, simulating real-world distractions. The bystander's role is most critical the Side Talk Rejection task, where their speech serves test for the assistant's ability to filter out device-directed input. Bystanders may also be present in other tasks, adding complexity the audio environment and challenging the assistant's speaker identification and intent recognition capabilities.
3.1.3 ACOUSTIC CONDITIONS
The dataset includes recordings from wide variety of environments, to capture the full spectrum of acoustic conditions encountered by wearable voice assistants.
Recordings are conducted in rooms of varying sizes (small, medium, and large), offices, and busy hallways. These settings introduce range of reverberation levels, back- ground conversations, and ambient noises such as air conditioning office equipment. Such condi- tions are particularly relevant for tasks like Search-Grounde Grounded QA and Closed-Boo QA, where the assistant must accurately process user queries despite potential acoustic interference.
Indoor Environments:
Outdoor and Mobile Scenarios: Sessions take place in parks, picnic areas, parking lots, cars, and near construction zones. These environments introduce dynamic background noises, including wind, traffic, and construction sounds, which can mask or distort wearer speech.
Noise Diversity and Signal-to-Noise Ratios: The dataset systematically varies the signal-to-noiss ratio (SNR) by including both quiet scenarios (e.g., soft whispers, rustling leaves) and high-noise situations (e.g., vacuum cleaners, subways, buses, motorcycles). This diversity ensures that the as- sistant's 's performance can be evaluated across a continuum from controlled, low-noise environments to highly challenging, real-world auditory scenes.
3.2
DATA COLLECTION
The process comprises three key stages: script collection, egocentric audio recording, and ground truth annotation. Each stage is carefully structured to maximize the realism, utility and reliability of the resulting dataset.
Script Collection The central objective in script collection is to ensure that the dataset authenti- cally represents real-world use cases. For spoken QA, we curate questions from the CRAG (Yang et al. 2024) and Head-to-tail (Sun et al. 2024) datasets, categorizing them into popular, static fac- tual questions for closed-book, QA, and long-tail, rapidly changing factual questions for search- grounded QA. For the remaining three tasks, first design several representative scenarios for each, then employ annotators to expand and construct multi-tune mulli-tuur conversations based on these sce- narios. For example, in the bidirectional speech translation task, seed scenarios typically involve
5

---

Published as a conference paper at ICLR 2026
foreigner approaching local to ask questions topics such finding locations, accommodations, transportation, and reservations. the tool calling task, we provide predefined tools including calendar, web search, local search, music player etc. Annotators, with the assistance of LLMs (e.g., Llama 3.3 70B), create multi-turn conversations based on these scenarios and domains.
Egocentric Audio Recording With the scripts prepared, the next step involves capturing egocen- tric multichannel audio data from glasses. To this end, we recruit a diverse group of native speakers: for the speech translation task, we hire native speakers of Italian, Spanish, Portuguese, German, and French who also understand English scripts; for the other tasks, we engage native English speakers. For each session, 2-3 individuals collaborate to simulate realistic interactions based on the provided scripts. Importantly, scripts serve as references during audio recording to enhance data quality; speakers are encouraged to follow the script loosely to ensure that the recorded speech sounds natu- ral and conversational, rather than read verbatim. Details are available in Appendix A.3
Ground Truth Annotation After data collection, we instructed our annotators to generate ground truth annotations for each dialogue. For the speech translation task, annotators transcribe the audio and provide corresponding translations based on the scripts and recordings. For the tool invocation task, annotators specify the appropriate API calls for each interaction. For spoken QA, primarily reuse labels from the original CRAG and Head-to-Tail datasets. For non-device-directe speech samples, assign special [Mute] token to indicate that these queries should be ignored as invalid.
3.3 DATASET STATISTICS
We collected 3,842 dialogues with egocentric multichannel audio recordings, comprising 547 Search-Grounded QA, 588 Closed-Book QA, 1,082 Side-Talk [ 1,125 Tool Calling, and 1,000 Translation tasks. Approximately | 31% of the recordings took place indoors, while 63% were recorded outdoors. In terms of noise conditions, 58% of the recordings were made in noisy envi- ronments, and 42% in quiet settings. A more detailed breakdown of environment and noise type distributions is provided in Appendix(A.4
BENCHMARKING
In this section, we systematically evaluate state-of-the-art SLLMs on the WearVox benchmark to assess their capabilities and limitations addressing the unique challenges of wearable contexts.
4.1
EXPERIMENTAL SETUP
4.1.1
BASELINES
We consider both proprictary and open-source models in our evaluation.
Open-Source Models: Gemma 3n TTem 2025a
Kimi-Audio
DDig et al. 2025
Qwen2.5-Omni (Xu et al.) (2025)
Proprictary Models: GPT-4o Audio Hurst et al. 2024), Gemini 2.5 Flash Comanici
al. 2025), GPT-5 w/ Whisper (OpenAl 2025)
Since the existing state-of the--at SLLMs are trained on single-channel audio, we follow previous work (Lin al. 2024) Xie et al. 2025) applying beamforming to convert the multichannel record- ings into single channel for evaluating single-channel SI.LMM performance.
4.1.2 EVALUATION SETTINGS
To facilitate comprehensive and consistent evaluation across diverse tasks, we divide our assessment into two settings: tum-based and dession-based evaluation.
"The Side Talk Rejection task contains 582 side talk and 500 valid queries duplicated from tool-calling tasks.
'Speech tramslation samples are excluded from this statistic due to missing audio metadata. http://wwwwmaaaapeaaoooo///////a pnevee preview

---

Published as a conference paper at ICLR 2026
Search Grounded QA
Closedbook QA
Tool Calling
Side Talk Rejection
Turn-based Micro-avg
Speech Translation
Baselines
Gemma 3n Kimi-Audio
29.4
20.4
5.7
59.9
29.7
14.8*
10.1
31.5
63
47.0
43.6
41.8*
Qwen2.5-Omni
35.8
29.8
7.3
60.4
33.1
43.9*
GPT-4o Audio
50.5
59.4
8.9
66.0
43.1
76.0
GPT-5 w/ Whisper
57.8
70.6
35.7
73.8
57.8
92.9*
Gemini 2.5 Flash
49.0
46.8
44.4
88.2
59.8
50.3
Gemini 2.5 Flash Thinking
48.8
61.4
68.1
91.4
71.3
70.1
Table 2: Benchmarking results for both open-source and proprietary SLLMS on four turn-based tasks (including the micro-average across all turn- based tasks) and session- based speech translation task. *Note: In the speech translation task, input audio for Gemma 3n, Kimi-Audio, and Qwen2.5 Omni is truncated at 30 seconds due to audio encoder context limits. In contrast, the GPT-5 Whisper baseline transcribes dialogues turn-by-turn using ground truth segmentation and diarization labels.
Turn-based Evaluation Turn-based evaluation is applied tasks such as Search Grounded QA, Closed-book QA, Tool Calling, and Side Talk Rejection, where model answer accuracy is assessed each turn. For Search Grounded QA and Closed-boed QA, we employ an LLM--ased judge that references annotated ground truth responses to evaluate answer quality. In the Tool Calling task, we utilize Abstract Syntax Tree (AST) evaluation, following the methodology described in Patil et al. (2024), to rigorously compare the structure and content of predicted tool calls. For Side Talk Rejection, performance is measured using binary accuracy, indicating whether the model correctly identifies and suppresses non-device- -irerreiiddde speech. Tool Calling and Side Talk task share the same task prompt as shown in Appendix [A.I] Thus, the model must generate a tool call for valid requests and produce special control token handle side talk.
Ression-based Evaluation Session- -based evaluation is used for the speech translation task to as- sess translation quality over entire dialog sessions. We provide ground truth translations and prompt an LLM judge to score each turn based on the quality of speaker diarization and translation. The final session score computed by averaging the turn-level scores, with penalties applied for missing or hallucinated turns. The LLM judge prompts and the session-level score aggregation function are detailed in Appendix A.2
LLM Judge Quality Validation The L-M-- judge for QA tasks was adapted from the auto- evaluation of CRAG, which has demonstrated over 98% agreement with human evaluation. For the translation task, validated the judge on 200 randomly sampled examples and observed strong Pearson correlation (r = 0.89) between our judge's scores and human ratings, with human raters using the same scoring scale the judge.
4.2
MAIN RESULTS
Table 2 reports the main results of WoxrVox, highlighting the substantial variability in performance across tasks and models. Open-source baselines, including Gemma 3n, Kimi-Audio, and Qwen2.5- Omni, generally underperform, particularly in search grounded QA and tool calling. This underper- formance can be partially attributed their relatively smaller model sizes (fewer than 8B param- eters), which limits their reasoning capabilities across different modalities, such as user audio and text context. In contrast, proprietary SLLMs demonstrate more balanced performance. Both GPT- 4o Audio and Gemini 2.5 Flash achieve overall scores above 40, with GPT-4o Audio excelling in speech translation (76.0%) and Gemini 2.5 Flash exhibiting strong robustness to side-talk (88.2%). However, observe that GPT-4o Audio occasionally ignores the task prompt and generates direct responses in the tool calling task, resulting in low tool calling accuracy (8.9%). We hypothesize that GPT-40 Audio is specifically trained to handle audio input and output, and that structured text output capability is not fully optimized. GPT-5 with Whisper achieves the best search grouned QA (57.8%) and closed-book QA (70.6%), yielding strong overall turn-based accuracy of 57.8%.
Enabling thinking mode in Gemini 2.5 Flash significantly improves performance on four out of five tasks, increasing overall turn-based task accuracy from 59.8% 71.3% and speech translation

---

Published as a conference paper at ICLR 2026
Task
Gemini 2.5 Flash
Gemini 2.5 Flash Thinking
GPT-40 Audio
Closedbook QA
1368.69
2287.76
1220.22
Search Grounded QA
1526.56
9194.94
1867.66
Speech Translation
2138.11
11321.49
7523 24
Side Talk Rejection
1306.62
2176.97
1341.00 .04
Tool Calling
1404.69
2084.19
1289.99
Table 3: Time to First Token (TTFT) breakdown per task for Gemini 2.5 Flash (with and without thinking mode) and GPT-4o Audio. All values are reported milliseconds (ms).
accuracy from 50.3% to 70.1%. However, it is important to note that thinking mode introduces sub- stantial latency, as extensive reasoning tokens are generated prior to producing the actual response.
In experiments, observed substantial increase in time to first (response) token (TTFT) latency for Gemini 2.5 Flash in Thinking mode, averaging 5546 ms compared to 1592 ms in non- Thinking mode. In Table we report the per-task TTFT breakdown for Gemini 2.5 Flash with and without thinking mode, as well as GPT-4o Audio. We observe that the thinking model exhibits sig- nificantly higher latency compared
its non-thinking counterpart, primarily due the overhead of thinking token generation. GPT-4o Audio demonstrates comparable latency to Gemini 2.5 Flash across most tasks, with the notable exception of speech translation, where the slower audio encoding during prefill likely contributes to increased delay. Overall, the latency difference between thinking and non- non-thinking model could significantly impact the user experience on wearable devices. Bal- ancing the trade-off between real-time responsiveness and response quality remains an important direction for future research.
4.3 CASE STUDY: MULTICHANNEL SLLMS
Beyond existing single-channel SLLMs, we present case study in which we develop multichannel SLLM and compare its performance to its single-channel counterpart. Our primary research question is: Does multichannel audio provide additional value over the beamformed audio channel in real-world wearable voice assistant tasks?
We construct our SLLM by building on Llama-4-Scout --ooutt-7----6 (Team] 2025b) and IB parameter Conformer (Gultii et al. 2020) speech encoder that pre-trained with BEST-RQ (Chiu et al. 2022 as in Ilama3 speech (Dubey et al. 2024). For training, we follow the speech alignment methodology described in AudioChatl.lama (Fathullah et al.) 2024). We begin with automatic speech recognition 16E to generate responses based on the corresponding
(ASR) data and prompt Llama-4-Scout-17 text transcripts. The original audio is then paired with these generated responses to create synthetic speech QA data. Both the speech QA and ASR datasets are used to train LLM, along with an audio feature projection layer, while keeping the speech encoder frozen. More implementation details are available in Appendix [A.6]
To enable native multichannel audio processing, convert all the original single-channel audio into simulated five-channel recordings, based on the microphone array configuration of Al glasses (Meta) 2024). Room impulse responses (RIRs) from real environments are used model spatial diversity. We further augment the data by adding indoor noise from a diverse corpus at random signal-to-noise ratios (SNRs) ranging from -5 dB 40 dB. Additionally, we introduce varying overlap ratios of bystander speech into the multichannel mixtures to simulate realistic acoustic conditions. We use the beamformed single-channel audio train a single-channel SLLM, which we refer to as SC Wearl (Single Channel Wearable Llama). In contrast, the model trained on multi-channel audio denoted as MC WearL.ama. As illustrated Figure unlike the SC Wearl which processes only the beamformed audio channel (c_x), the MC WearLlama processes both channel 0 (c.0), typically the channel with the highest SNR, and the beamformed channel in an interleaved manner.
compares SC WearLlama and MC WearLlama on the Wear Vox benchmark. MC WearLlama shows clear improvement in tool calling (63.9% vs. 58.5%) and side-talk rejection (93.9% vs. 85.4%), indicating that spatial audio cues help the model better separate user-directed speech from background interference and execute device-control tasks more reliably. However, both models
8

---

Published as a conference paper at ICLR 2026
Channel 0
G
MC Wearllama
Response
Audio Encoder
Mic
Multi channel
vs
Amey
Aurlio
Beamformer
SC Wearllama
Resporsee
Figure 2: Illustration of SC WearLlama and MC WearLlama inference. SC WearLlama encodes only the beamformed audio channel (c,x), whereas MC WearLlama processes both channel 0 (c O).ypicclylly the channel with the highest SNR, and the beamformed channel in an interleaved manner.
0.7
Inviromennt
Indears Cuodeors
0.7
Notse Level
0.4
Accunccy
Genema
AotiiAutto
Guend Grod Good
GP
Gendin 2.5 Path
Geealel 3.8 ruse mestin
Carcds Whisger
Wetriiane
Stsetlemee
Ganont 30
Gameló 2.5 Rast
Gonint Resh Thonzop
C0T- ad Whigper
SC Woeerterno
MC teafumm
Figure 3: Effect of acoustic environment on SLLM performance in turn-based tasks.
exhibit nearly identical performance on the two QA tasks. We hypothesize that the advantages of multichannel audio diminish when recordings are made primarily in quiet indoor environments. Further discussion the impact of acoustic environments can be found in Section 4.4
4.4 IMPACT OF ACOUSTIC ENVIRONMENTS
Figure compares model performance turn-based tasks across different acoustic environments, specifically contrasting indoor versus outdoor and quiet versus noisy conditions. The results reveal clear trend: most models exhibit degraded performance in outdoor and noisy environments. For example, Gemma 3n, Qwen2.55-mmni GPT-4o, Gemini 2.5 Flash, and GPT-5 w/Whisper all ex- perience performance drops ranging from 3% to 15% in outdoor settings. with Gemma showing the largest degradation, likely due to its smaller model size. In contrast, Kimi-Audio demonstrates significantly higher accuracy in outdoor environments, which can likely be attributed its pre- training data (Ding et al.) 2025) that includes a balanced mix of noisy and clean audio. Interestingly, the reasoning model Gemini 2.5 Flash Thinking exhibits strong noise robustness, with accuracy in outdoor noisy conditions matching or even slightly surpassing its performance in indoor quiet environments. This suggests that robust to real-world noise. Similar trends are observed for both SC WearLlama and MC WearLlama. which are trained with noise-augment augmented audio. Notably, MC WearLlama demonstrates significantly greater robustness to outdoor noise, achieving approximately 5% higher accuracy in outdoor noisy environments compared to SC WearLlama, while maintaining comparable performance in indoor quiet conditions. These findings address the research question posed in Section 4.3) indicating that
speech- languu-a models are inherently more

---

Published as a conference paper at ICLR 2026
Search Grounded QA
Closedbook QA
Tool Calling
Side Talk Rejection
Turn-based
Baselines
Micro-avg
WearLlama MC WearLlama
43.3
42.5
58.5
85.4
43.3
42.2
63.9
93.9
66.4
Table 4: Evaluation results of SC WearL.lma and MC WearL.lama on turn-based tasks.
multichannel audio enhances the noise robustness of SLLMs in real-world wearable voice as- sistant tasks.
CONCLUSION
In this work, we introduced Wear Vox, the first comprehensive benchmark specifically designed to evaluate the performance of voice assistants in realistic wearable scenarios. Through diverse set of multi-channel, egocentric audio recordings collected via Al glasses, WearVox captures the unique challenges posed by wearable devices, including environmental noise, motion artifacts, and the need to distinguish device-directed speech from background conversations. Our benchmarking of state- of-the-art proprictary and open-source SLLMs reveals that current real-time models struggle with these challenges. particularly in noisy outdoor environments, highlighting the gap between existing solutions and real-world requirements and point out an important research direction on the trade-off' between real-time responsiveness and response quality in reasoning SLLMs. Furthermore, our case study demonstrates that leveraging multi-channel audio inputs can significantly improve model ro- bustness noise and enhance the ability to discriminate between device-directed and background speech. These findings underscore the critical role of spatial audio cues developing context-aware, reliable voice assistants for wearable devices. We hope that Wear Vox will serve as valuable source for the research community, driving the development more robust and intelligent wearable AI systems that seamlessly integrate into everyday life.
Future Work Several promising directions remain for extending WearVox. First, incorporating recordings from diverse hardware platforms with varying microphone array geometries would en- able evaluation of model transferability and reduce device-specific optimization concerns. Second, extending the benchmark to include multimodal signals-such as visual data from cameras and mo- tion information from IMU sensors-woul better reflect real-world wearable computing. Visual cues can aid speaker identification and object grounding, while IMU data can help disambiguate device-directe directed speech through head orientation and gesture detection. Finally, expanding task cov- erage to include simultaneous translation, proactive assistance, and multi-step planning would fur- ther advance wearable Al capabilities. We believe these extensions will establish WearVox as an evolving platform that continues to drive progress in context-aware voice assistants.
REFERENCES
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishray Chaudhary. Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras arXiv preprint arXiv:2503.01743, 2025.
Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Hefferman, John Hoffman, al. Seamlessm4t: massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308, 11596, 2023.
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby Tan, and Haizhou Li. Voicebench: Benchmarking Ilm-based voice assistants. arXiv preprint arXiv:2410.1 17196, 2024.
Chung-Chenn Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning randomm-rooeecion with random- projection quantizer for speech recognition. International Conference on Machine Learning, pp. 3-15-3924. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626. 2025.
Alexandre Défossez, Laurent Mazaré, Manu Orsini. Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. arXiv preprint arXiv:2410.00077 2024.
Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504. 18425, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Ma- hadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Audiochatlama:: Towards general- purpose speech abilities for 1lms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol- ume 1: Long Papers), pp. 2522-5533 2024.
Anmol Gulati, James Qin, Chung- Saogg- Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmennt transformer for speech recognition. Interspeech 2020, 2020.
Will Held, Michael J. Ryan, Aditya Shrivastava, Ali Sartaz Khan, Caleb Ziems, Ella Li, Martijn Bartelds, Michael Sun, Tan Li, Woody Gan, and Diyi Yang. Cava: Comprehensive assessment of voice assistants. https: : //tggggtubb..////////////...hhtt com/ SALT CELLT-NLP// /CAVA 2025. URL lhtps::/ talkarena. wwwoor/gcaa... cava A benchmark for evaluating large audio models (LAMs) capabilities across six domains: turn taking, instruction following, function calling, tone awareness, safety,
and latency.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 1xXiv:2411.151244 2024.
Chia-Hsuan Lee, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee. Spoken squad: A study of miti- gating the impact of speech recognition errors listening comprehension. In Proc. Interspeech 2018, pp. 3459-3463, 2018.
Guan-Ting Ting Lin, Shih- Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, and Hung-yi Lee. Full- duplex-bence v1. 5: Evaluating overlap handling for full-duplex speech models. arXiv preprint arXiv:2507.23119, 2025.
Ju Lin, Niko Moritz, Yiteng Huang, Ruiming Xie, Ming Sun, Christian Fuegen, and Frank Seide. Agadir: Towards array-geometyy agnostic directional speech recognition. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 11951-11953 IEEE, 2024.
Ray Ban Meta. Ray ban meta. http::/wwwww....accooaoaocceeeea 2024.
OpenAl. Gpt-5 system card. hutp://hgeeuicccoonndee/g/gsssss 2025.
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and
Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Advances in Neural Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 14492-28518. PMLR, 2023.
Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19166 2024.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable are large language models (Ilms)? aka will Ilms replace knowledge graphs? In NAACL- HLT, 2024.
Gemma Team. Gemma 3n. http://uuigoogglldee////mmmd///s
2025a.
Llama Team. The llama 4 herd: The beginning of new era of natively multimodal ai innovation. 2025b.
https://ai.mets https://ai.mta.coombbboo http://wwwwo.//ooooommmoo/oooooo
Bandhav Veluri, Benjamin Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. Be- yond turn-based interfaces: Synchronous Ilms as full-duplex dialogue agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2024.
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, Aiti Aw, and Nancy Chen. Audiobench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp-
4297-4316, 2025.
Yijing Wu, SaiKrishna Rallabandi, Ravisutha Srinivasamurthy. Parag Pravin Dakle, Alolika Gon, and Preethi Raghavan. Heysquad: A spoken question answering dataset. arXiv preprint arXiv:2304. 13689, 2023.
Jiamin Xie, Ju Lin, Yiteng Huang, Tyler Vuong, Zhaojiang Lin, Zhaojun Yang. Peng Su, Prashant Rawat, Sangeeta Srivastava, Ming Sun, al. Thinking in directivity: Speech large language model for multi-talker directional speech recognition. arXiv preprint arXiv:2506, 210Xi::200..4977. 14973, 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, al. Crag-comprehenssie rag benchmark. Advances in Neural Information Processing Systems, 17:10007-11999 2024.
Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, and Yuexian Zou. End-to-end spoken conversational question answering: Task, dataset and model. In Findings of the Association for Computational Linguistics: NAACI. 2022, pp. 1219-1232, 2022.
Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. Salmonn-omni: A codec-free Ilm for full-duplex speech understanding and generation. arXiv preprint arXiv:2411.18133 2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612 2024.
Zihan Zhao, Yiyang Jiang, Heyang Liu, Yu Wang, and Yanfeng Wang. Librisqa: A novel dataset and framework for spoken question answering with large language models. IEEE Transactions on Artificial Intelligence, 2024.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 TASK PROMPTS
Listing Closedbook QA task prompt
You are given audio question. Your task is to answer the question in few words as possible.
Listing 2: Search Grounded QA task prompt
You are given audio question, which was asked at (query time). Your task is to answer the question in as few words as possible. You are also provided with the references below, which may or may not help answer the question.
(search [search_result]
Listing 3: Speech Translation task prompt: English- example
**T'ranslate a Two-Person Conversation Audio**
You will given an audio input a conversation between two people, speakero and 'speakeri " Translate their conversation in real-time according to the following rules:
'speaker0' speaks German. Translate their dialoque to **Englishxy. 'speaker1' speaks English. Translate their dialogue to **German**.
**tutput Format:**
Provide only the translated text.
Retain the original speaker labels ("speakerO', 'speakerl Use the following format:
'speaker0; [Translated text in English]
'speakerl: [Translated text in German
Examppe Translation::.
Audio Input:** { Conversation between 'speakerO' and 'speaker!
'speaker0: Hast du die Flugtickets gebucht?
'speaker:: Yes, have booked them for tomorron. Translated Output:**
sspakerr; Have you booked the flight tickets?' 'speakerl: Ja, ich habe sie fur morgen gebucht.
Listing 4: Tool Calling and Side Talk Rejection task prompt
You are friendly AI voice assistant
the smart glasses. Sometimes, the user may need to interact with people around them may longer want to engage in the conversation with the AI assistant. In such cases, answer with [Mute]'. Otherwise, follow the instructions below.
Tools
You have access to the following tecls. You might need to use one or nore functions/toolls calls fulfill the task. If none needed, then proceed the response.
You can call tools using the syntax:

> [["name:: laool__ame__oo_ "parameters"; ("<argl>"; .... "<arg2 >": ... ("name": pool__nme_bbr>, "parameters"; ("<argi>"; .... " arg2>": +]14//0000//>>>
defondiiii
Info
Instruction: You are a helpful AI assistant designed facilitate user interaction with wearable device.
13

---

Published as a conference paper at ICLR 2026
The current time is (current_time The user is currently in {current caazrent__lcation)
A.2 LLM JUDGE
We use Llama 3.3 70B LLM judge, the QA and translation judge prompts are provided in Listing 5 and 6. For session-level score aggregation in translation task, given the list of LLM judge scores per turn: {21,82,.... ....,,,.............. ,8) ). we compute the session based score as:
1= 8
==
max N, NGT)
Where is the number of turns predicted by model and NGT the number of turns from ground truth annotation.
Listing 5: Prompt for speech translation LLM judge.
E=Eallatin a Live Two-Way Translation Model with Time -ware Scoringy*
Your task to assess the performance of two-way live translation model in translating conversations between. two speakers, **aakke
((lang speakerr) translated (lang speaker1]) and **Speaker1 {f translated Saagg_seakerr0
You will evaluate the model's translation quality for specific fic ground truth turn, considering the prior turns and the model's output translations.
** Innut Data**
You will receive the following input data:
Ground truth translations for each turn, including timestamps. (translation Iround truth}
2. The target ground truth turn to be evaluated, including timestamps (target_tur__rrond_ttttee
3. Prior turns of the target ground truth turn from ground truth translation includi ng timestamps. (prior turns ground truth)
4. Full translations from model's cutput, without timestarps. [model_outputs
**Groundtruth Format**
The ground truth format appears to be text file containing conversation between two speakers. The format is as follows:
[speaker ID] [timestamp] [utterance]
Where:
'speaker ID': number (0 or 1) indicating which speaker is speaking. 'timestamp" : pair of numbers in square brackets, representing the start and end times of the utterance in seconds (e.g.
1[16.16,20 20.46]
'utterance"; the text of what the speaker said. Speaker 0's utterances are ALMAYS in (lang speakerl), and Speaker i's utterances are ALMAYS (lang_speakero).
*Model Output Format**
The model output format appears to be text file containing conversation between two speakers, with each 1ine representing single turn in the conversation. The format is as follows:
'speaker [ID]: [utterance]
Where:
14

---

Published as a conference paper at ICLR 2026
"peaaker[DD] string indicating which speaker is speaking, with 'ID' being either 0 or 1.
uuterannce" ; the text of what the speaker said, which is a translation of the original text. Speaker 0's utterances SHOULD ALWAYS BE { lang speakerl), and Speaker I's utterances SHOULD ALNAYS BE in sana_ere
Evaluation Steps**
To evaluate the model's performance, follow these steps:
**Step 1: Align the Model Output with the Ground Truth**
Match the model output turns with the corresponding ground truth turns based on the order, speakez label, prior turns, and ground truth timestamps.
Maintain the sequence from the ground truth timestamps.
If the model output turn is missing the ground truth or the speaker label is incorrect, assign score of **0.00**.
**St 2: Fine-Grained Translatien Quality Scoring**
Compare the aligned model output turn with the target ground truth turn:
1. Speakee Check++: the speaker label is incorrect, assign score ..0.00**
2. LLnguuae Check**: Speaker 0's utterances should ALMAYS sang_speaker), and Speaker 1's utterances should ALMAYS be in the Language is incorrect, assign
of 3. **Mannigg and Accuracy Evaluaticnes: Assess the model's output translation against the ground truth translation, considering:
Lang_speakerO] +00.00**
* M*eaning preservations. (full semantic equivalence) *Comleeteness (no cmissions or unnecessary additions) **Tone/ttyee* (formality, politeness, etc.)
++raamarr & fluency**
4. **'li Re-laated Adjustments**:
the ground truth duration adjust
the score:
**Long duration overly short translation penalize for under-transiation
*wshort duration + overly long translation penalize for
possible hallucination
If the translation is unrelated to the time-bound content,
assign a score of **0.00**
5. Strrict Fine-Grained Scoring Scale*t: Assign score between **0.00** and ++1.00,,, in increments of **0.05**, based the evaluation criteria.
** Scooiin Scale**
Use the following scoring scale:
..1.00. Perfect match meaning, tone, grammar, and length
**0.9.**: Trivial synonym/word-ordee, differences, perfect meaning
++0.9+++: Very minor rewording, full meaning intact
**0.85**: Slightly less natural phrasing, meaning intact
..0.80+4; Minor grammar/style issues but meaning preserved
mild omissions .*0.65-0.55**: Noticeable meaning loss mistranslation of & detail
7*0.75-0.70*x: Small meaning shifts
..0.50-0.40wx. Major omissions or additions, significant meaning loss
**0.35-0.20**: Large distortion of meaning, wrong tense, or unrelated
info
..0.15-0.05**: Almost completely wrong; only tiny fragments correct
.+0.00+*: Wrong speaker, missing turn, totally unrelated, or
hallucinated content
Important Notes**
Be **strict**: even small meaning changes reduce the score.
Missing turn or wrong speaker label ++0.00++. Time-based penalties are applied for unrealistic translation length vs. speech duration.
15

---

Published as a conference paper at ICLR 2026
Prioritize **meaning fidelity and temporal alignment***
**Output *ormat**
Strictly output only the following JSON format: No additional words or sentences outside of H and } of the json format since the output will be parsed by a python script.
((*translation_scorts score": 0.00-1.00, "reason"; "reason for score", trrggeed_ttr__mmmeeooorr
"aligned turn from model output without timestamp", starget_ture ttt_r_ttuunnnnnnnneeeeeeee trranddsutthttaassee "target turn from ground truth translation without timestamp" }}
Listing 6: Prompt for QA LLM judge.
Assume you are human expert in grading predictions given by & model.
You are gi a quest on and a model prediction. Judge if the prediction matches the ground truth answer by following these steps:
1: Take it as granted that the Ground Truth is always correct.
2: should Prediction indicates it is not sure about the answer, "score"
"0"; otherwise, go the next step.
3: If the Prediction exactly matches the Ground Truth, "score" is 1.
4: If the Prediction does not exactly match the Ground Truth, go through the fol following steps and likely give score as 0.
5: If the Ground Truth is number, "score" is 1 if and only if the Prediction gives a nunber that almost exactly matches the ground truth.
6: the Prediction is self-contradictory, "score" must be 0.
7: the prediction
not answering the question, "score" must be 0. 8: the prediction "score" is
a concise and correct summary of the ground truth
9: It ground truth contains set of items, prediction must contain exactly sare items for the score to be 1.
10: otherwise, "score" is 0.
### Output a JSON blob with an "explanation" field explaining your answer as short as possible and an "score" field with value or 0. You should make the judgment based on provided examples. Examples:
Question: "which company has higher eps, btu or cma?"
Ground Truth: "ema"
Prediction "it is not possible to determine which company has higher eps."
Output: ["score": 0, "explanation"; "The prediction is not sure about the answer.")
(34 more examples)
Question: (query) Ground truth: (ground truth)
Prediction: (prediction)
16

---

Published as a conference paper at ICLR 2026
30
C0
400
120
120
150
-150
Figure 4: Ego-centric audio recordings, with conversational partners positioned between - -60% and 60% Bystanders may speak from any angle.
Wearer
Conversation
bystander
Background Noise
Volume
Origin, facing to
_330",000000 (*) [1m, 1.5m) distance
Medium
recording
Origin, facing to
1380" 0*l with [tm, 1.5m) distance
No
Small
572 ullerence
0
Origin. facing to o"
#+60",+30".00 [1] with [1m, 1.5m) distance
No
No
Medium
572 racarding
0
None
Yes, slanding from [x150',
Yes
Large
0
572 recording
Origin, facing to 0
1120" ,a90" 1800 _a30), with 2m, 3m) distance
None
Yes, standing from (=150", 1120" .190" _200 .330), with 3m)
No
Large
572 recording
Origin, facing
Origin, 1.5m)
Ouldoor
572 recording
Origin, facing
Shopping
recording
Total recording
1.7 recording
2,288 recording
(30% indoors, in Quier No Background)
(70% Outdoors in Noay Environment)
Figure 5: Audio Recording Distribution
A.3 WEARVOX AUDIO RECORDING
Participants are positioned in different locations and distinct environments, and speak aloud using provided scripts (commands queries) as references. As shown in Figure the wearer dons glasses cquipped for audio recording, while the conversation partner and bystanders are placed at various angles relative to the wearer. The distribution of recordings is illustrated in Figure
A.4 DISTRIBUTION OF ACOUSTIC ENVIRONMENTS
Figure 6 and illustrate the distribution of audio recording location and noise type. Speech transla- tion samples excluded from this statistic due missing audio metadata.
17

---

Published as a conference paper at ICLR 2026
Large Room Area 16%
Large Room Area
Park Nature Area
37%
27%
Small Room Area
Modium Roum Araa
25%
Indoors 31%
Outdoors 69%
Busy Halway
cum
sween oer pemee
toma Arr-to Hallway
Brank Krea
Cary Broak ama
Sboat Crosuss
6%
6%
Office/ / desk
6%
de
11%
Figure 6: Audio Recording I Location Type Distribution: Approximately 31 % of the recordings took place indoors, including various recording rooms, hallways, and cafes. The remaining 69% were recorded outdoors, such as in parks, streets, and parking lots.
Conversations (1m)
Quiet Office 52%
Vehicies (15m)
30%
21%
Noisy Loud 58%
Quiet 42%
Consiructinn Hoisa (15m) 17%
pusting Num
4%
Wind
Comernatons (1m)
3%
Rustiing Leaves 30%
Somei Chaniig
12%
Other 8%
Music 7%
Figure 7: Audio Recording Noise Type Distribution: Approximately 57% of the recordings were made in noisy environments, featuring sounds such as vehicle noise, music, and bystander speech. The remaining 43% were recorded in quiet settings, where background noise such rustling leaves was negligible.
A.5 MODEL PERFORMANCE BREAKDOWN PER ACOUSTIC ENVIRONMENT
We provide detailed per-noise-type breakdowns for all leading models in the Table Wind noise consistently degrades performance across all systems. GPT-4o, Gemini 2.5 Flash (non-thinking). and SC WearLlama exhibit particularly severe degradation under construction noise, while Gemini 2.5 Flash Thinking and MC WearLlama demonstrate greater robustness.
A.6 SC AND MC WEARLLAMA IMPLEMENTATION
Model Architecture Both SC and MC WearLlama are built on Llama-4-Scuut-17B-BE (Team) 20256) and a 1B parameter Conformer (Gulati et al. 2020) speech encoder. Similar to Llama Speech (Dubey al. 2024), audio encoder operates at sampling rate of 12.5 Hz, converting every 80ms audio into one audio embedding. As illustrated in Figure & for MC WearLlama, the same encoder is applied to both channel and the beamformed channel, and the generated audio em-
18

---

Published as a conference paper at ICLR 2026
GPT 4o
Gemini 2.5
Gemini 2.5 Flash Thinking
GPT
SC WearLlama
MC WearLlama
Noise Type
#Samples
Audio
Flash
w/ Whisper
Construction Noise (15m)
47.2
74.0
52.0
58.9
70.1
Lawnmower (Im)
41.2
64.7
79.4
61.8
70.6
70.6
68
Loud Singing
45.0
67.5
80.0
60.0
72.5
82.5
40
70.8
67.0
47.2
66.0
67.9
Normal Conversation (Im)
42.2
63.5
71.1
55.6
63.0
66.5
630
Other
37.1
66.5
72.5
55.1
68.3
75.4
167
Quiet Office
68.1
76.9
63.0
68.4
68.4
Quiet Whisper (Im)
28.1
56.1
77.2
45.6
61,4
64.9
Rustling Leaves
41.7
54.3
71.9
59.4
59.1
65.3
470
Vacuum Cleaning
38.0
49.3
70.4
50.7
62.0
69.0
Vehicles (15m)
40.3
62.9
73.9
61.9
64.9
69.9
402
Wind
35.7
28.6
40.5
57.1
23.8
33.3
42
Table 5: Per-noise-tyre breakdowns for all leading models. Wind noise consistently degrades performance across all systems. GPT-4o, Gemini 2.5 Flash mnontthinking) and SC WearLlama exhibit particularly severe degradation under construction noise, while Gemini 2.5 Flash Thinking and MC WearLlama demonstrate greater robustness.
FLO
MC
Response
Channel 0 **
Wearllama
f,X
0.0
RU0
Shared weights
Channel X
SC Wearllama
Response
Audio Encoder
C
C
Figure 8: Illustration of SC WearLlama and MC WearLlama inference. SC WearLlama encodes only the beamformed audio channel (c x), whereas MC WearLlama processes both channel (c 0), typically the channel with the highest SNR, and the beamformed channel in interleaved manner.
beddings are interleaved and input to the Llama-4-Scond 4-ccuut decoder along with embeddings. The decoder is trained to generate text responses by conditioning on both audio and text representations.
Training Data Our training data is curated from multiple sources, including: (1) pseudo-labeled ASR data as described ScamlessM4T (Barrault et al. 2023); (2) Speech QA data generated from ASR audio described in AudioChatLlame (Fathullah et al. 2024); and (3) additional Speech QA data converted from text ceeruuctionfollowwiii datasets (Lambert et al. 2024) using our in- house TTS system. Both ASR and Speech QA data formatted Text In, Speech In, Text Out problems following the formulation ST) > To, where T₃ is the system prompt, S₁ is the audio input described in the previous section, and To is the text output. For the ASR task, To is the transcript; for Speech QA, To is the response. Note that no Wear Vox data samples are used in model training.
Multichannel Audio Augmentation To train MC WearLlama, we convert all the single-channel audio into simulated five-channel recordings, based on the microphone array configuration of AI glasses (Meta 2024). We simulate the multi-channel data by convolving with real-recorded room impulse responses (RIRs), and adding noise and sidetalk random signal-to-nois ranging from -5 dB to 40 dB. Formally, we have S, = * + n + + x * h3, where is the user speech, is the noise sampled from our in-house noise corpus, and is the sidetalk. 11, h2,
ratios (SNRs)
19

---

Published as a conference paper at ICLR 2026
Experiment Setting
LibriSpeech test-other WER
Train: Czi Test: Ca
8,58%
Train: Cz and Co: Test: co and 61
8.21
Train: Cg and co: Test: ca and
8.16%
Train: Oz and Co: Test: C₂ and cu
7.38%
Table 6: MC Wearl microphone arry generalization experiment. We tested different combinations of simulated audio channels during inference on unseen channel combinations on the simulated LibriSpeech test-other set.
h3 are the real-world multi-channel RIRs which are measured and collected from different rooms by covering various distances and directions.
Training Objective We train the model using the standard next-token prediction objective. The supervised fine-tuning loss is defined as:
SSF
log P(C S1, (2:;0))
where is the length of the output sequence To 1 20 :2....... +2 to represents the i-th token in the output text, 12 denotes all preceding output tokens, and 0 represents the model parameters. The model is optimized to minimize the negative log- -igliiel of the ground-truth output text conditioned on both the input text prompt I7 and the input speech signal S₁.
A.7 TRANSFERARILITY OF MC WEARLLAMA ON DIFFERENT MICROPHONE LAYOUTS
Our MC WearLlama processes two channels: Channel 0 (highest SNR) and the beamformed chan- nel. This design is relatively aggnostc-agnosicc compared to approaches that explicitly model all microphone positions. In Table 6 we tested different combinations of simulated audio 5 channels during inference and found that while testing on unseen channel combinations leads to some perfor- mance degradation, the
model outperforms its single-channel counterpart.
20


Reviews:
[{"review_id": "inC7zFGWDy", "reviewer": "Reviewer_muyQ", "summary": "WearVox introduces a wearable-specific benchmark of 3842 egocentric multichannel recordings collected with AI glasses, spanning 5 speech tasks, with each clip having rich environment and position metadata for nuanced analysis. The paper benchmarks open-source and proprietary SLLMs and reports large drops in noisy/outdoor conditions.", "questions": "- Will you release a public leaderboard and evaluation server? If so, please provide the planned URL and timeline.\n- Please report per-task latency and time-to-first-token (TTFT) for each model—especially in ‘thinking’ mode—so we can judge real-time feasibility on wearables.\n- Could you please eleberate more on the MC Wearllama? The description is not clear.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First benchmark aimed squarely at wearables with egocentric, multi-mic audio, diverse indoor/outdoor scenes, and explicit side-talk; prior suites largely miss these factors.\n- Both open and proprietary SLLMs; headline finding: most real-time SLLMs land ~29–59% on WearVox, highlighting difficulty.\n- Five tasks with clean input/output definitions.\n- Support multi-channel processing, for testing the SLLMs.", "weaknesses": "- Some reporting is aggregate. More per-environment/per-distance breakdowns (beyond the figures) would make failure modes easier to act on. \n- \"Thinking\" boosts scores but increases TTFT substantially, this deserves heavier emphasis for wearables.\n- A careful proofreading pass is needed to improve clarity as well as typos.\n- No examples to listen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9YPvruNxy", "reviewer": "Reviewer_Nq5Q", "summary": "This paper introduces WearVox, a benchmark dataset designed to evaluate voice assistants in realistic wearable scenarios, e.g. with wearable glasses. The authors mention that existing voice assistant benchmarks fail to capture challenges of wearable, egocentric audio such as motion, wind noise, background and side-talk rejection. This paper provides a benchmark dataset of 3,842 recordings captured using AI glasses in multi-channel audio format for five diverse tasks that AI assistants may have to do: Search-Grounded QA, Closed-Book QA, Side-Talk Rejection, Tool Calling, and Speech Translation. The authors study existing voice assistant methods, Speech Large Language Models (SLLMs) as baselines and also introduce specialized new models that handle native multichannel audio better.", "questions": "1. More details on SC/MC WearLlama would be nice, especially the training data and methods. Was there any overlap or fine-tuning done on WearVox? You demonstrated that proprietary models struggled with noise. How did your noise augmentation strategy for Wearllama differ from the standard techniques you assume are used in commercial models?\n\n1. Is there a way to measure the \"realness\" and diversity of the scripts? Particularly humans are bad at creating entropy, what measures were takes to make sure sure there is enough diversity?\n\n1. Are there plans to make the WearVox dataset, or at least a sample of it, publicly available or browseable? An interactive way to explore the audio and metadata would be invaluable for the research community.\n\n1. You used an LLM to judge the QA and Translation tasks. What steps did you take to validate the LLM-judge itself? For instance, how did its scores correlate with human expert scores on a subset of the data?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper is a solid contribution to benchmark real-world AI assistant applications in a wearable setting, such data is very hard to find and is expensive to manually curate, script, and collect. It is a first of a kind dataset for this emerging setting in HCI and human-AI interfaces.\n\n1. The baselining is done for a wide range of commercial models, and is done across settings and tasks that matter for this wearable setting.\n\n1. \"Side-Talk Rejection\" is arguably the single most important and overlooked challenge for an always-on wearable assistant. Evaluating this explicitly is a huge contribution.\n\n1. The paper carefully curates data, using specific angles and distances, and capturing audio across 13 different noise types, the paper has created a dataset that reflects the structured complexity of real-world interactions, not just random noise.", "weaknesses": "1. For the custom trained models, it is not clear if there is data leakage? It is also not clear from the paper what data was used to train these models. Presumably the baseline models were also trained with noise augmentation as is standard with commercial-grade speech models.\n\n1. There doesn't seem to be an easy way to explore the benchmark. Perhaps this is a way to prevent leakage, but it would be great to be able to explore this dataset in an interactive way.\n\n1. It is hard to get a sense of diversity in the dataset. Presumably the annotators who were asked to write multi-turn conversations did a good job, but there are no quantitative evaluations about the _quality_ or diversity of this dataset. Additionally, the limited scale might not capture edge cases that matter like speaker accents.\n\n1. The data was collected using the microphone array from a specific set of AI glasses, and does not capture the diversity in hardware.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXUyYTzZ6l", "reviewer": "Reviewer_tmBp", "summary": "The paper presents WearVox, an egocentric multimodal dataset that integrates head-mounted RGB video, binaural microphones, and body-contact microphones to capture synchronized, real-world audio-visual data for wearable perception tasks. To complement this dataset, the authors propose WearVoxNet, a unified multimodal encoder that jointly processes visual and acoustic signals for tasks such as speaker localization, active speaker detection, audio-visual event detection, and grounding. The overarching motivation is to advance embodied perception research by enabling models that can understand and respond to multisensory inputs from the perspective of a wearable device, bridging the gap between egocentric vision and human-centered audio perception.", "questions": "(1) Would it be possible to include a new set of experiments demonstrating cross-dataset generalization (for instance, training on WearVox and testing on Ego4D or AVD) to validate that the learned audio-visual representations transfer effectively to unseen wearable contexts and real-world data?\n(2) Can the authors present a focused study on the contact microphone’s contribution—analyzing placement robustness, ambient noise resilience, and task-wise performance gains—to highlight its distinct sensing value and clarify why it is essential for reliable egocentric audio understanding?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) WearVox represents a significant and original data contribution. It is, to the best of current knowledge, the first large-scale dataset that combines egocentric RGB video, binaural recordings, and contact microphone signals in naturalistic scenarios. This unique combination allows for studying wearable perception in realistic settings, including interactions and self-generated sounds that traditional datasets cannot capture.\n(2) The dataset’s coverage of multiple downstream multimodal tasks—including speaker localization, active speaker detection, audio-visual event detection, and grounding—demonstrates its broad utility. Such diversity supports a range of research directions in speech perception, ambient understanding, and diarization, positioning WearVox as a potentially foundational dataset for wearable multimodal intelligence.\n(3) A key technical insight is the inclusion of contact microphones, which capture body-conducted vibrations and help distinguish self-speech from surrounding voices. This modality provides an innovative solution to one of the core challenges in wearable audio—disambiguating the user’s voice from ambient speech—making the dataset especially valuable for applications like personal assistants, AR/VR communication, and cognitive hearing aids.\n(4) The data collection emphasizes ecological validity by recording in diverse real-world environments and activities. This diversity strengthens the dataset’s relevance for real-world deployment and ensures it captures the dynamic acoustic and visual variability encountered in wearable scenarios.", "weaknesses": "(1) The WearVoxNet model, while competent, does not introduce fundamentally new architectures or fusion strategies. It primarily builds upon established audio-visual encoder paradigms, combining features through standard cross-attention or concatenation methods. As a result, its novelty lies more in the dataset than in methodological innovation.\n(2) The paper does not explore cross-dataset generalization, such as training on WearVox and evaluating on existing benchmarks like Ego4D or AVD. Such experiments would be essential to demonstrate the robustness and transferability of the learned representations beyond the curated data, highlighting whether WearVox-trained models can generalize across wearable domains.\n(3) While the contact microphone modality is a central innovation, its contribution is not rigorously analyzed. The paper lacks systematic ablation studies on factors such as sensor placement, environmental interference, and failure modes. A deeper examination of when and how this modality enhances performance would provide greater insight into its reliability and sensitivity across contexts.\n(4) The work does not address practical deployment constraints. Wearable devices typically face stringent limitations on power consumption, latency, and computational capacity, yet the paper does not discuss whether WearVoxNet—or any of its variants—can be efficiently adapted for on-device inference or lightweight architectures suitable for real-world use.\n(5) The empirical comparisons could be stronger. Benchmarking against state-of-the-art models from recent audio-visual or egocentric perception challenges (such as AVE, Ego4D, or AVD) would better substantiate the claims of high performance and clarify where WearVoxNet stands relative to the broader literature.\n(6) Finally, while the contact microphone clearly boosts self-speech detection, it is not clear how it affects other multimodal tasks. A more detailed analysis of modality contribution across different perception scenarios—such as non-self speech, ambient sounds, or silent visual events—would strengthen the interpretability of the findings and clarify when this modality provides genuine cross-task benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5YXdvMZOZB", "reviewer": "Reviewer_6q8y", "summary": "This paper collects a large set of in-the-wild data of users interacting with expected Voice AI tasks using microphone enabled glasses. The range of tasks tested is Search-Grounded QA, Closed-Book QA, Side-Talk Rejection, Tool Calling, and Speech Translation. For each task, there are multiple possible speakers including the wearer, the conversational partner, and bystanders and speech from each should be handled differently on a contextual basis. The authors first benchmark a range of current audio models on these benchmarks, then do an ablation of their own custom models designed for this data type and find that multi-channel signal processing is key to handling noise and acoustic cues in the data.", "questions": "- Are there any ablations for the SC and MC Llama on how much more specified MC Llama is to the specific layout of the microphones in these particular glasses? Could you train MC Llama on a simulations of a different Multi-Channel layout and get similarly strong transfer to WearVox? While it makes sense that multi-channel is useful, it's not clear to what degree a single channel model might be more transferrable across devices.\n\n- Are you able to share the participant demographics mentioned in the ethics statement? It says \"demographic data was aggregated and used solely for fairness analysis.\" but I wasn't able to find such analysis and it would be valuable to share.\n\n(Typo on 347. Appedix should be Appendix)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The dataset covers realistic tasks, realistic environments, and realistic challenges for speech models. Despite being primarily focused on this realism, the benchmark is clearly difficult for frontier speech LLMs especially open-weights speech LLMs. It will clearly help measure (and therefore drive) progress.\n- The multi-party nature of the environment is a welcome addition to the speech benchmarking space which usually operates on data from relatively clean single-speaker environments. With this, the benchmark will clearly drive models to be more robust to the natural challenges from real multi-party environments (such as distractions and interruptions) that often plague today's speech LLMs.\n- The authors then go above and beyond reporting the results of their benchmark by showing an example of a low-level ablation which the benchmark allows them to enable the impact of.", "weaknesses": "- The benchmark seems somewhat likely to incentivize models to optimize for the specific hardware the benchmark was recorded on. This is of course true of any speech benchmark which uses a consistent recording device, but might become increasingly impactful for devices which have specific form factors and layouts for their microphone arrays.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)