ClarifyVC: Clarifying Ambiguous Commands in Vehicle Control with a Hybrid Data Augmentation Pipeline
Authors: Hange Zhou
Zhonglin Jiang
yingjie cui
Mingzhe Zhang
Xiaotang Wang
Hengwei Dai
Qiyao Yu
Yong Chen
Yongqi Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Interactive Control Systems
Clarification-First Dialogue
Ambiguity Resolution
Hybrid Data Augmentation
Function-Calling Language Models
Human Validation and Robustness
Primary area: datasets and benchmarks

Abstract: Natural language interfaces for vehicle control must contend with vague commands, evolving dialogue context, and strict protocol constraints. 
We introduce ClarifyVC, a unified framework that integrates a hybrid data-augmentation pipeline (ClarifyVC-Data), reference models trained on the data (ClarifyVC-Models)
and a evaluation protocol (ClarifyVC-Eval). 
The agent-orchestrated pipeline generates diverse, ambiguity-rich dialogues from real-world seeded queries under schema and safety constraints, while the evaluation protocol systematically probes single-turn parsing, conservative clarification under extreme fuzziness, and multi-turn grounding. 
Fine-tuning on ClarifyVC-Data yields consistent gains—up to 15\% higher parsing accuracy, 20\% stronger ambiguity resolution, and 98\% protocol compliance—across realistic in-cabin scenarios, with human-in-the-loop assessments confirming high realism, coherence, and applicability. 
ClarifyVC thus advances beyond simulation-only datasets by tightly coupling real-world grounding with scalable generation and standardized evaluation, and provides a generalizable pipeline for broader interactive control domains.

Published as a conference paper at ICLR 2026
CLARIFYVC: CLARIFYING AMBIGUOUS COMMANDS IN VEHICLE CONTROL WITH A HYBRID DATA AUGMEN- TATION PIPELINE
Hange Zhou
Zhonglin Jiang?.3.
Yingjie Cui2..
Mingzhe Zhang'
Xiaotang Wang
Hengwei Dai
Qiyao Yu²
Yong Chen 3
Yongqi Zhang Li
1 The Hong Kong University of Science and Technology (Guangzhou)
2 Nanjing University of Information Science and Technology
Geely Automobile Research Institute (Ningbo) Co., Ltd.
4 The University of Hong Kong
ABSTRACT
Natural language interfaces for vehicle control must contend with vague commands, evolving dialogue context, and strict protocol constraints. We introduce Clari- fyVC, a unified framework that integrates a hybrid data-augmentation pipeline (ClarifyVC-Data)) reference models trained the data
and evaluation protocol EllrriyyCC-Evar The agent-orchestrated pipeline generates diverse, ambiguity-rich dialogues from real-world seeded queries under schema and safety constraints, while the evaluation protocol systematically probes single-turn parsing, conservative clarification under extreme fuzziness, and multi-turn ground- ing. Fine-tuning on Clarify VC-Data yields consistent gains-up up to 15% higher parsing accuracy, 20% stronger ambiguity resolution, and 98% protocol compli- ance across realistic in-cabin scenarios, with human-in-the- confirming high realism, coherence, and applicability. Clarify thus advances beyond simulation-only datasets by tightly coupling real-world grounding with scalable generation and standardized evaluation, and provides a generalizable pipeline for broader interactive control domains. Our code and dataset are available at:
assessments
INTRODUCTION
Natural language interfaces are becoming cornerstone of interactive control systems, from au- tonomous vehicles (Wen et al. 2024) to smart homes (Thukral et al. 2025), robotics (Sikorski et al., 2023). and other embodied agents (Bick al. 2024). These systems require the ability interpret vague instructions, maintain multi-turn dialogue context, and execute actions under strict protocol constraints. In the automotive domain, the rise of autonomous vehicles has already transformed human-machine interaction, making natural language commands crucial for intuitive and trustworthy control of hundreds of onboard functions (Zheng et al. [2024] Wang et al.) (0024)). However, Vehicles face pervasive ambiguity- user commands are often vague, protocol mappings incomplete, and existing evaluation metries inadequate (Ma et al.) 2024). Traditional intent detection and slot-filling methods perform poorly under ambiguity and context shift, while current benchmarks lack realism, coverage, or failsafe metrics (Chun al. 2025).
Public perception reflects these gaps: 58% of individuals feel uneasy about in-car voice assistants which is a key part of autonomous driving stack, and 25% express complete distrust in their relia-
"Equal contribution.
Corresponning author. Email: @ hw..tgge.euu.cnn

---

Published as a conference paper at ICLR 2026
bility (Wenskovitch et al. 2024; Peng & Shang 2024). A core reason is that current LLMs, though strong in general reasoning, struggle in safety-crtical control (Brahman et al. (2024). They hallucinate under ambiguous instructions, fail to request clarifications when uncertain, and lack strict protocol adherence in task orchestration (Dai et al. 2024). These weaknesses are compounded by the absence of high-quality, reality- datasets and standardized evaluation protocols, limiting progress toward reliable in-vehicle dialogue systems (Nguyen et al. [024 Zou et al. (024). To close this gap, we introduce ClarifyVC, unified framework for clarifying ambiguous commands in vehicle control. It integrates a hybrid aata-auumennation pipeline (Clarify VC-Data), reference models trained on the data (Clarify (l-MolyyCCMMdells and a three-tier evaluation protocol (Clarify SerrifyC--EBe VC-Eval) to evaluate the data quality and model performance. Our work targets the safety-critical task of mapping natural-language in-car voice commands- ambiguous or under-specified-ing into strictly validated, sshema-aligned function calls, requiring accurate parsing, conservative ambiguity detection, and multi-turn parameter grounding.
At the core of Clarify VC-Data is hybrid augmentation pipeline seeded from over 20k authentic in-vehicle commands drawn from a proprietary corpus of 4M+ production-level interactions. Through structured ambiguity injection, adversarial perturbations, and multi-tury clarification, the pipeline synthesizes ambiguity-rich yet
samples that target robustness and safe execution. The resulting dataset, Clarify has been validated through human evaluation and distributional alignment experiments, demonstrating close correspondence to real-world usage patterns. Fine- tuning LLMs on this data yields an average 15% improvement in parsing accuracy, underscoring the pipeline's 's practical value for safety-critical language interfaces.
Beyond dataset construction and training, we introduce ClarifyVC-Evar a three-tier evaluation protocol that explicitly targets real-world ambiguity in function-call tasks (Jing el al.) [024bb Chao al.) 2024: Wu et al. 2024) Jiang et al.] (024a). ClarifyVC-Evar VCaEval plays a dual role: (i) audits the benchmark itself liseti-testing whether the data is realistic and ambiguity-rich- and (ii) it provides a unified lens to evaluate model capabilities
semantic parsing, execution fidelity, and safety compliance. By jointly assessing data validity and functional reliability, the protocol addresses key gap in prior work, which typically isolates dataset realism from model accuracy and thus misses their interaction in safety-critical settings. To operationalize the data-side audit, we additionally define Dataset Quality Score (DQS) that aggregates ambiguity diversity (AD), protocol compliance (PC), and realism (R). Together, ClarifyVC-Eval and DQS constitute comprehensive, scalable framework for auditing datasets and benchmarking models under realistic ambiguity.
Extensive experiments demonstrate that Clarify VC substantially improves performance in control tasks. Fine-tuned models achieve 15% higher parsing accuracy, 20% better ambiguity resolution, and 98% protocol compliance, while also reducing inference latency by 30% compared to baseline systems. Additional ablation studies confirm the necessity of each module in the data augmentation pipeline, yielding the best trade-off between diversity, coherence, and adherence. Multi-run evaluations further validate robustness, showing consistently low variance (<1 x)) and statistically significant improvements across metrics. Human-in-the--lee annotators rating generated dialogues highly on realism, coherence, and practical applicability. Together, these findings highlight VC as reliable and efficient framework for robust language understanding under real-world ambiguity in vehicle control. In summary, our contributions are threefold:
assessments corroborate these results, with expert
1. CI Framework: A unified framework for clarifying ambiguous commands in vehicle control and interactive systems. It integrates a hybrid data pipeline, a three-tier evaluation protocol, and reference models, offering end-to-end standard for safe and deployable language interfaces.
2. ClarifyVC CC-Data&Moode A hybrid,
and human-validated dataset built from 20k+ real-world seed commands, expanded with controlled fuzziness and adversarial variants. By training on the high-quality data, we release reference models that show consistent gains in accuracy, clarification, and safety compliance.
3. ClarifyVC-Eval A three-tier evaluation protocol that disentangles under-specification, ambiguity clarification, and multi-turn grounding, along with a Dataset Quality Score which ensures the benchmark aligns with real-world distributions and maintains high-quality standards. By explicitly targeting these failure families, the protocol enables comprehensive and safety-awars assessment of function-call understanding, addressing gaps left by conventional single-turn accuracy.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 METHODS FOR CLARIFYING AMBIGUITY AND MULTI- TURN COMMAND PARSING
Natural-Language command understanding has progressed from structure-aware parsers to end- to-end LLM solutions for mapping utterances to executable actions (Zheng et al., 2024 Wang
et al. 2024a). Early pipelines emphasized
intent/slot structures and hierarchical modeling (Sriram et al. 2019; Wang et al. 2024b) Okur et al. (002)). More recently, LLMs have
been applied to enable direct intent grounding. rule translation, and task formalization (Shao et al.
2024 Choudhary et al.
2024 Manas & Paschke (023).
In the domain of vehicle visual command understanding, datasets like Talk2Car (Deruyttere iet al. 2019). CI-AVSR (Dai et al., 2022a). and doScenes (Roy et al. 2024) provide real-world instruction-action pairs and visual grounding contexts, but primarily support single-turn mapping rather than interactive clarification. Beyond single-turn parsing, logical disambiguation methods such
as LogicalBeam (Bhaskar et al. 2023) have been explored, while other datasets and studies (e.g. CHAMBI) highlight cross-cultral or spatial ambiguity challenges (Zhang et al. 2024b] Saparina & Lapata 2024). Frameworks for task decomposition and retrieval-augmented decision making further support complex instruction following (Shen et al.) [024] Yang et al. 2024). Parallel streams examine unimodal parsing (Zhang et al. 2024a). synthetic data generation (Liu et al.) 2024), and distillation for instruction following (Ding et al. 2024), alongside
command datasets (Liu et al.,
2023 Li et al. 2024). Together, these approaches offer modeling, training, and data-centric tools for tackling ambiguity and multi-turn semanties in command parsing.
2.2 LIMITATIONS OF EXISTING APPROACHES AND OUR POSITIONING
Despite steady progress, three gaps persist. (I) Ambiguity management and uncertainty signaling. Many systems lack explicit mechanisms to detect under-speciffiattio, trigger clarifying questions, expose calibrated confidence, which is critical in seneti-se interaction (Pramanick et al. 2022 Wenskovitch et al. 2024 Lee et al.] 2024). (2) Evaluation scope. Benchmarks often emphasize synthetic distributions (Zhang et al., 2024a Liu et al. 2024 2023 Li et al. 2024), offering limited coverage of multi-turn grounding and protocol-aware execution; task-centric frameworks (Shen et al., [024| Yang al., (222)) provide only partial Instruction-ggeneaation distillation pipelines (Ding et al. 2024) rarely tie ambiguity real logs enforce function-call
single-turn parsing, narrow modalities,
visibility into clarification behavior. (3) Data realism and compliance. Instruction generation and
protocols, hindering transfer to deployed systems.
Clarify addresses these gaps with a unified framework that: (i) couples a hybrid, real-log seeded sedeeee augmentation pipeline with controlled adversarial/fuzzy evolution to surface realistic ambiguity; (ii) introduces three-tier evaluation protocol that disentangles single-turn parsing, clarification under extreme fuzziness, and multi-turn grounding with execution checks; and (iii) reports reference models trained on the data with protocol-aligned metrics. Our data generation pipeline, benchmark, and evaluation protocol demonstrate strong generalization and practical utility in real world settings. Beyond in-cabin voice control, the framework's 's clarification strategies and compliance-oriente evaluation naturally extend to broader domains of human machine interaction, including smart homes, medical dialogue, and embodied intelligence, where safety and interpretability remain critical.
METHODOLOGY
We instantiate Clarify VC as a unified framework comprising a hybrid data-augmentation pipeline reference models trained on the data CCariifVC--Molde and a safety-aware Seeded with 20k+ authentic in ve-icle commands (drawn from 4M+ production logs), the pipeline expands queries via structured ambiguity injection, adversarial perturbations, and multi-tume pullti-tur&a clarification under protocol constraints. The resulting corpus is validated through human studies and distributional alignment with real-world usage.
three-tier protocol (ClarifyVC-
3

---

Published as a conference paper at ICLR 2026
Stage Adversanal
Seed query
I'm roasing
my back
degree
Stage Fuzzy
AGA
Set
FIA
Constraint
aggregation
Set
Process flow
command
Stage 3. Mult-turn Evolution
conditonino.
Fuzzified command
Set
cold adjux conditioiig
MEA
ClarifyVC-Data
Do want to
too not, aduet the or nonditonns.
Adiaed the condiiiniing
cond
bveen lowared by degrane
temparature you Pleace deculled infoamettio..
Multi featti-turn dialogue
How seculd you aduanlud?
Figure I: ggent-orchestrrtedd stage-wise generation flow. A schema-constrained pipeline executes se- mantic parsing, adversarial construction, fuzz injection, and multi-turn evolution on commands synthesize ambiguity-rich single- and multi-turn dialogues under protocol constraints. The resulting corpus forms Clarify VC-Data: a hybrid, realism-ale aaliimm- suman-validated benchmark with standardized function-call annotations and broad ambiguity coverage.
3.1 CLARIFYVC-DATA: AGENT-ORCHESTRRATRR PIPELINE
Stage-wise pipeline. We adopt agent-orchestratee modular pipeline with four stages: Semantic Parsing Module (SPA), Adversarial Generation Module (AGA), Fuzz Injection Module (FIA), Multi- Turn Evolution Module (MEA), each implemented with prompt-engineered, pre-trained LLMs without task-specific fine-tuning. Specifically, SPA/FIA/ME. are implemented with DeepSeek-R! (API-based) for semantic parsing, fuzz injection, and multi-turn dialogue evolution, while AGA is realized with @wen2.5-721 5-72B (via vLLM) to perform protocol-constrained adversarial rewriting. These choices combine scalability and strong substitution. The total compute cost of synthesizing the benchmark remains modest, as generation relies primarily on API calls and lightweight orchestration:
while ensuring modularity for drop-in
SPA parses each seed command into E, P) standardized grounding.
AGA produces syntactically valid yet ambiguous variants Cady under protocol constraints.
FIA converts Cadv into softer fuzzed instructions e' (parameter omission, subjective modifiers, mild distortion); both tiers retained.
MEA expands e' into coherent multi-tun turti dialogues D for long-horizon grounding.
The adopted sequence (SPA + AGA - FIA + MEA) is not arbitrary but empirically validated. We conducted controlled ablations that permuted the order or removed individual stages. Evaluation results (Table show that the default order achieves the best balance across ambiguity diversity, dialogue coherence, and protocol adherence. For example, reversing FIA and AGA substantially reduced diversity, while removing either stage markedly degraded ambiguity coverage. This confirms that the chosen order is optimal for synthesizing realistic yet challenging interactions.
This yields hierarchical pool: (1) SPA+AGA Cadvi (2) +FIA
c': (3) +MEA + D. To
encourage diversity while preserving operational validity, each sample is scored by
Q(c) H(c) a) T[ec protocol-compliant). a 0.6,
(1)
4

---

Published as a conference paper at ICLR 2026
where (e) is ambiguity entropy and II(-) indicates compliance (Appendix B.2
Reference models ClarifyyC-Modelyy We obtain Clarify VC-Modlls by supervised fine-tuning
open-source backbones (e.g., LLaMA3-8B, Qwen2.5-7B/72B DeepSeek -i-bcck----isttllledd ClarifyVC: Data with schema-aligned function-call targets, using reacher-forees cross-entropy objective and JSON-schema ceNNsshhema-cnnstranneee decoding inference(Experimenn settings be seen in Appendix [B). Training is performed with early stop on a delayed test split and evaluated on a separate 2k test set, averaged 5 random seeds (std.,< 1%). We release the Qwen2.5-7B-SFT checkpoint and training/evaluation configs. Notably, while larger backbones (14B, 32B, 72B) show strong results, the 7B model achieves the best trade-off between accuracy and computational efficiency, reducing inference cost by an order of magnitude while delivering comparable or superior performance under our protocol. As will be shown Table 2 the models fine-tuned with ClarifyVC- consistently surpass the zero-shot base models in different scenarios.
3.2 CLARIFYVC-EVALL EVALUATING DATASET QUALITY AND MODEL PERFORMANCE
In this part, we introduce Clarify VC-Eval to evaluate the quality of Clarify VC-Data and Clarify VC-- Models. A Dataset Quality Score (DQA) is used to ensure the benchmark aligns with real-world distributions and maintains high-quality standards. Meanwhile, three-tier evaluation protocol that disentangles under-specification, ambiguity clarification, and multi-turn grounding, is proposed to evaluate the generation quality of different LLMs.
Automated quality and human validation for Dataset. We summarize dataset quality using the Dataset Quality Score (DQS):
DQS λ1 AD A2 PC A R, A2,, (0.4.0.3,0.3).
(2)
where AD, PC, and R are three complementary measures capturing ambiguity coverage, schema correctness, and realism of recovery pathways. We select (0.4, 0.3,0.3) 0.3) via grid search maximize Spearman correlation with human ratings while preserving consistent ranking across baselines. Full sweeps are provided in Appendix C.4]
The exact definitions of AD, PC and R are provided below.
Ambiguity Diversity (AD). AD quantifies how well the dataset covers the five major fuzzy categories (intensity, boundary, entity, mode, referential). Let A be the set of ambiguity types and p(a) the empirical proportion of type a A. We compute:
AD KL(p(a) u(a))
log [A]
where ua) is the uniform target distribution. Higher AD indicates more balanced ambiguity patterns, avoiding collapsed or single-type dominance.
Protocol Compliance (PC). PC measures the proportion of samples whose gold function calls satisfy the production HMI schema and rule-engine constraints:
PC Σ [[c E SSchema A of Ssatety:
where Sychema denotes JSON action-slor validity and Ssatery captures type, range. and dependency rules. PC reflects structural correctness of the dataset's functional targets.
Realism (R). R measures the alignment of each sample with real in-cabin interaction patterns. For each instruction Fú, we retrieve the most similar real-log utterances (ar[²) (using embedding similarity) and check whether its gold call of matches the dominant action-slot pattern in retrieved logs:
1
Σ
Ife Mode((c(a)) (1)(a()))))))))
R
(
therefore evaluates whether dataset samples resemble historically observed user behaviors rather than synthetic artifacts.
5

---

Published as a conference paper at ICLR 2026
Usec: "Iacceuse temperature.
User: too bot"
Inference "lemperuture incrased C'C.'
you adjust conditioning rilldww the windows?
Clarification Puth. temperature increase would you peodar?
Usen djust the air conditioning
Single-Round Fuzzy Instruction Parsing
"Plazee proviée the target trmperatur and windsppood"
Usee: "Tun switch"
User: "22 degrses, medium wind"
"Could you specify switch you want
"Olar, the air conditioning has been act to 22"C, wind speed geaes.
(ii) Extreme Fuzzy Instruction Counter-
Multi-round Dialogue
Figure 2: Illustration of Clarify (i) parse mildly fuzzy commands into precise function calls, (ii) adopt safe clarification under extreme vagueness, and (ii)) sustain multi-round dialogue for coherent, grounded execution capturing ambiguity, safety, and interactivity in real-world control.
Three-Tier Protocol for Comprehensive Model Evaluation. While dataset validation secures distributional realism, robust model assessment requires an evaluation protocol that can reveal failure families invisible single-turn accuracy. To this end, when evaluating models, Clarify ClarifyVC--BB VC-Eval operationalizes three complementary tiers of evaluation: (i) Single-round fuzzy instruction parsing, which tests the model's ability parse mildly ambiguous commands by disambiguating challenges such as under-specified parameters, vague references, and subjective expressions; (ii) Extreme fuzzy instruction quustterruetiiinnn. examining whether the model adopts safe clarification strategies when confronted with severe ambiguity, specifically its capacity to detect extreme uncertainty and ask relevant clarifying questions; and (iii) Multi-turn dialogue, assessing the ability to address challenges like multi turn dependency and memory, which is crucial for iteratively recovering missing semantics, maintaining dialogue and parameter coherence, and executing the accumulated commands reliably. The protocol ClarifyVC-Evar as illustrated in Figure 2 spans single-turn parsing, clarification under extreme fuzziness, and multi-turn turn dialogue grounding, addressing gaps in existing evaluation metrics and enabling more realistic assessment of safety, robustness, and interactivity in control-oriented language interfaces
VC-EEal consists of three complementary tiers, each probing distinct aspect of interactive rehicle-control reasoning. Tier evaluates semantic parsing under underspecification, using intent- and function-level metrics (IRA, PEP, IHR, FHR) assess fine- fine-grained correctness of grounded calls. Tier measures safe clarification under extreme fuzziness via FDR, CQC, and PCR, testing whether models detect ambiguity, avoid unsafe guesses, and follow interaction protocols. Tier assesses ongg-hriizon multi-turn grounding, evaluated by DC, FESR, and parameter completeness to ensure coherent dialogue and reliable execution across turns. Full metric definitions are provided in Appendix C Together, these tiers cover the key decision points required for safe, robust function-call interaction.
Rationale and scope. Our analysis of 20k+ real-world in-vehicle logs shows that failures cluster into three families: under-speciffcation, insufficient clarification, and long-horizon grounding. These are precisely captured by ClarifyVC-Eve which not only measures success rates but also tracks protocol violations, yielding diagnostics that better reflect operational safety. Importantly, the same decision points recur across broader HCI and embodied intelligence, making the protocol directly transferable.
4 EXPERIMENTAL
We conduct extensive experiments to address the following research questions: RQ1: What's the quality of Clarify SerrffyVC-DD VC-Data evaluated under DQS in Equation 2 and suman-grounder validation? RQ2: How well do existing LLMs handle complex and ambiguous vehicle control instructions? RQ3: Does fine-tuning on Clarify RQ4: How accurately can LLMs execute structured function calls under protocol constraints? RQ5:
improve model performance in realistic command understanding?

---

Published as a conference paper at ICLR 2026
Table Comprehensive quality evaluation of Clarity (addresses RQ1: dataset quality). Automated metrics (left) benchmark ClarifyVC-DDen against prior datasets; human validation (right) reports detailed scores for ClarifyVC-DDe under a blind, 5-point Likert protocol. Comparative
human ratings for other datasets are provided in Table 7
(a) Comparison with baselines using four metrics.
(b) Human validation (ClarifyVC-Data, VC-Data only)
Dataset
AD
PC
R
DQS
Aspect
Score
Agreement
Talk2Car
0.50
0.85
0.60
0.62
Linguistic Realism
4.6 0.2
93%
doScenes Roy et
2024
0.56
0.81
0,64
0.65
C1-AVSR DDa el al.
20226
0.53
0.82
0.61
0.64
Ambiguity Plausibility
4.6 - 0.1
96%
DeepSeek Distilled
0.55
0.80
0.65
0.65
Dialogue Coherence
4.7 + 0.2
94%
GPT-o1 Distilled
0.60
0.82
0.70
0.69
Qwen2.5 Distilled
0.58
0.78
0.68
0.67
Practical Applicability
4.5 + 0.3
91%
LLaMA3 Distilled
0.62
0.80
0.72
0.70
Clarify VC-Data
0.89
0.95
0.82
0.88
Can open-source models, when properly tuned, match or surpass proprietary models in vehicle control tasks? The complete experimental setup, including the experimental environment and the agent-orchestrated pipeline used to generate the evaluation test sets, provided in Appendix
4.1 EVALUATION ON THE DATA QUALITY (RQ1)
As mentioned in Section 3.2, we introduced DQS and human validation assess the quality of the dataset. Table D () shows that Clarify exceeds previous datasets and distilled baselines on all four automated axes.
For human- uuman-grounded validation, we sample 500 dialogues from Clarify VC--Dty Data and ask five indepen- dent annotators to each dialogue 1-5 Likert scale along four dimensions: linguistic realism, plausibility of ambiguity, dialogue coherence, and practical applicability. Ratings are collected under blind, shuffled protocol where annotators do not know which dataset a dialogue comes from. As summarized in Table bb), ClarifyVC-Data achieves high average ratings (4.5-4.7/5) with strong agreement 91-96%), confirming that the generated benchmark both natural and practically usable.
To further ensure objectivity, we also conduct blind comparative human study against three public datasets (Talk2Car, doScenes, and Glaive-fc-v2), using the same protocol and scales. The detailed cross-dataset comparison is reported in Appendix and Table Together, these results confirm that Clarify not only surpasses prior datasets on automated metrics but also passes stringent numan-groundie validation, ensuring both scalability and real-world applicability.
4.2 BENCHMARK TEST ON BASELINE MODELS (RQ2, RQ3)
To rigorously assess the necessity and effectiveness of Clarify VC-Data, we conduct a two-part empirical study centered benchmarking the instruction-folloiing capabilities of LLMs in vehicle control scenarios. In the first part, we evaluate four representative models, including Qwen2.5-72B, LLaMA3-70B, Claude 3, and GPT-4, under a zero-shot setting across five benchmark datasets, in- cluding three open instruction-followinn datasets (Talk2Car, CI-AVSR, doScenes), one programmatic function-call dataset (APIGen), and our proposed ClarifyVC-Data. The results in Figure reveal a consistent performance drop on VC-Data highlighting its higher linguistic complexity, ambiguity diversity, and multi-turn reasoning demands.
In the second part, to evaluate the effectiveness and generalizability of Clarify VC-Data, we conduct systematic experiments across twelve open-source I.LMs, including Qwen2.5 (0.5B-72B) Team 2024c), LLaMA3 (8B, 70B) (Team] 2024b a). and DeepSeek-RI Distilled (1.5B-70B) (DeepSeek-Al 2025). Each model is evaluated under three settings: Zero-shot (ZS); inference without adaptation; Few-shot (FS): inference with 4 in-context examples; SFT: supervised fine-tuning Clarify
Data. Training details and loss definitions are provided in Appendix B The results demonstrate that while pre-trained models exhibit limited capabilities under zero- and few-shot conditions, fine-tuning on Clarify VC-Data yields significant gains across all metrics, especially in ambiguity resolution and multi-turn coherence. Together, these findings underscore the practical difficulty of realistic vehicle
7

---

Published as a conference paper at ICLR 2026
Table 2: Multi-run evaluation on Clarify under the three evaluation settings introduced earlier (ZS / FS/ / SFT) (addresses RQ3: effectiveness of Clarify MarrfyVC-based fine-tuning). Entries are means over 5 independent runs; per-cell standard deviation columns (Appendix B B.4). For readability we report means; the "Max (pp)" row summarizes the largest observed standard deviation in each column. All models are evaluated on held-out test set of 5k instructions, separately constructed to differ in distribution from the 20k training corpus, ensuring fair generalization assessment.
1.0 percentage point (pp) across all
Single-Round Accuracy SFT
Fuzzy Detection Rate
Multi-Turn Consistency
Model
ZS
FS
ZS
FS
SFT
ZS
FS
SFT
Max a (pp)
$0.7
<0.8
<0.6
<0.8
50.8
<0.7
<0.8
<0.7
50.8
Qwen2.5-0.5B
59.2
64.5
75.1
57.0
60.3
73.5
54.8
59.1
72.4
Qwen2.5-1.5B
63.4
67.2
78.9
60.5
64.3
76.4
58.9
63.0
74.1
Qwen2.5-3B
66.9
70.3
81.6
64.0
67.5
79.1
61.4
64.8
77.3
Qwen2.5-7B
74.3
77.1
89.0
72.0
74.8
87.6
70.2
72.5
85,4
Qwen2.5-
78.1
79.5
91.3
75.3
76.9
89.2
73.0
74.2
88.0
Qwen2.5-32B
80.8
81.6
93.1
78.2
79.0
91.5
75.5
76.0
89.6
Qwen2.5-72B
82.5
88.4
95.8
81.0
83.2
93.6
79.8
82.9
92.3
LLaMA3-8B
72.0
74.3
87.1
70.0
72.8
85.3
67.3
69.0
83.0
LLaMA3-70B
81.2
80.1
79.0
77.4
92.5
76.5
75.3
90.8
DeepSeek-RI -IDtttiillee
66.0
70.1
83.7
64.3
62.0
65.0
DeepSeek-RR R1-Disiilled-SB
74.5
76.3
73.7
69.4
71.0
83,9
DeepSeek-BR S1-Distilled-700
82.4
84.0
94.8
80.1
81.9
93.1
78.0
80.2
command understanding and establish Clarify VC-Data as a high-fidelity benchmark for developing and evaluating robust instruction followoinnnnooowiingggooo models.
Beyond Lingle-datsed fine-tuning, we additionally test cross datsset generalization to rule out distributional bias. Two backbones are fine-tuned on six public datasets and evaluated on an unbiased 4k open benchmark, disjoint 2k real-log set, and 1K fuzzy OOD suite. As shown in Appendix C.4 (Table 29). Clarify VC-Dats achieves the highest accuracy across all conditions, confirming that its effectiveness stems from data quality rather than overlap with evaluation distributions.
We assess model performance on all three tiers of the benchmark which explained section 3.2. As shown in Table 2 several trends emerge: (i) SFT consistently outperforms ZS and FS across all models; (ii) FS offers marginal gains over ZS, particularly for small-scale models; (iii) large models (c.g., Dwwe2..5-2BB), FS occasionally underperforms ZS, likely due to prompt truncation sub-optimal context bias. These findings highlight the limitations of in-context prompting and validate the effectiveness of ClarifyVC-Date VCtaata fine-tuning benchmark for vehicle command comprehension.
Table 2 shows consistent gains across heterogeneous backbones, and Figure highlights Clarify VC- Data's s higher ambiguity and richer structural complexity. Together, they provide mechanistic and empirical evidence that the improvements arise from the dataset's 's quality rather than confounding factors. We further analyze this causal relationship: Appendix C for details.
4.3 EVALUATION OF BASIC INSTRUCTIION CAPABILITIES (RQ4)
To assess the fine-grained function execution ability of LLMs in vehicle control scenarios, we compare twelve representative systems spanning both open-source 2-B.:/..557///1//32/////
LLaMA3- 8B/70B, DeepSeek- RR1) and proprietary models (GPT-4 (OpenAl et al. 2024) with function calling, GPT-4o (OpenAl, 2024a), OpenAI-ol (OpenAI 2024b)) Doubao (Doubao Team 2025). Claude 3 (Anthropic] 2024)). All models are evaluated under a unified function-call setting on four key metries: Intent Hit Rate (IHR), which measures whether the correct intent identified; Function Hit Rate (FHR), which checks whether the predicted API/function matches the gold standard; Parameter Completeness (F1-Score), which evaluates the accuracy and coverage of slot/parameter filling: and Protocol Compliance Rate (PCR), which assesses whether generated function calls adhere to predefined API schema and safety constraints. Further details of metric definitions and evaluation procedures provided in Appendix C In all four metrics, the "gold standard" function call for each instruction is defined by a unified validation pipeline: (i) exact semantic match to the expert- expert-annotated intent and slot values under our 37-action, 146-parameter schema, (ii) protocol and safety compliance
8

---

Published as a conference paper at ICLR 2026
Owen2.5-722
LLaMA3-70B 70B
Claude 3
GPT-
Intent Accuracy
100
Function Accuracy
200
Parameter Coverage
MutiiTum Surcess
Scor (4))
Scon
Scono
8
70
70
10
66-
Cheese
andrones
Clada
dedevve
Arron
cundpre ood
Cheeea
AFRINS
Ceadprc Doo
mazzor
andernes
ettim
cualyc tea
casdye
Figure 3: Zero-shot evaluation of four representative LLMs across five benchmarks (addresses
RQ2: model robustness under complex/ambiguous instructions). We compare the performance of Qwen2.5-72B, LLaMA3-700 Claude 3. and GPT-4 on four core metrics across five datasets: Talk2Car, CI-AVSR, doScenes, APIGen, and Clarify VC-Data. Results show that while all models perform well on existing benchmarks, they exhibit notable drop when evaluated on Clarify VC-Data For instance, GPT-4's intent accuracy drops from 92.5% on APIGen 70.5% on Clarify VC-Data, and its multi-turn success rate drops from 77.5% to 61.7%. This highlights the increased difficulty and real-world alignment introduced by our benchmark.
Incent Hit Rate (HH)
Runction Hit Rato (PHR)
F1-Score
Protocal Comply Rate (8.7L-18.7
Score (1)
Scare (5)
(%)
Scome (%3
deees ar cross
sloeet
Qweet court
Oweet
Figure 4: Comparative performance of 13 LLMs the Clarify across four critical function- call metrics (addresses RQ4: protocol-constranne execution accuracy). All values are reported in percentage (%). Our finetuned model (Qwen2.5-7B-SF7 achieves state-of state-of-the-art results across all metrics.
with the production HMI rules, and (iii) for ambiguous cases, majority-vote agreement from blind human adjudication. Formal rules and examples are provided in Appendix C
The evaluation is conducted on a test set of 4,000 control commands, comprising 2,000 curated samples from the Talk2Car dataset and 2,000 from real-world in-vehicle control logs, covering diverse command types such as lighting, HVAC, navigation, and media operations. These metrics reflect both semantic accuracy and system safety compliance, which are critical in production-gran automotive systems.
As shown in Figure our VC-idel (Qwen2.5-7B-SFT more ablation studies can be seen in Appendix D). fine-tuned on VC-rffyC--aiiiee consistently achieves state-of-the-as performance across all evaluation dimensions. Notably, it surpasses leading closed-source models such as GPT-4o and Claude 3 in both execution correctness and safety alignment, demonstrating the impact of targeted comain--pecific fine-tuning. This confirms that instruction-tune LLMs benefit substantially from high-quality control-oriented supervision when deployed in structured vehicular environments.
4.4 EVALUATION ON ADVANCED SCENARIO (RQ5)
Building upon foundational function-call evaluations, we assess the same 13 prominent large language models under complex, ambiguous, and multi-turn vehicle control scenarios to test instruction- following capabilities under realistic conditions.
We utilized three test sets each with 5,000 examples (15,000 samples in total), which are fully disjoint from all training data: (1) single-round fuzzy instruction parsing, (2) extremely fuzzy instruction counter- nuntee-questtoning (requiring clarification), and (3) multi-round dialogues with evolving contexts. These benchmarks non-overlapping seed commands and distinct logs ensure striet

---

Published as a conference paper at ICLR 2026
Qwen2 5-728
1LaMA3-108
GPT4
Claate
Qwen2 5-1T-rTTjpun)
Safetyy crtiica
Entertainment
Autonomous Driving
Comfort
C3C
CDC
CDC
CDC
Figure 5: Comparison of LLM performance across four realistic vehicle-control scenarios (addresses RQ5: advanced OOD and multi-turn scenario evaluation). In every scenario, the polygon for Qwen2.5- 7B-SFT fully encloses the others, demonstrating state-of-ther offthe-art accuracy, coherence, and safety compliance with a much smaller parameter footprint.
out-of-distribution evaluation (Appendix A.1). Models were evaluated across six metrics: Intent Recognition Accuracy (IRA), Parameter Extraction Precision (PEP), Fuzzy Detection Rate (FDR), Quutte-- Coverage (CQC), Dialogue Consistency (DC), and Final Execution Success Rate (FESR). Detailed descriptions of the test sets and their generation process are provided in Appendix
ClarifyVC-Model (iven2.5-B-SFFT) consistently outperformed all tested models across all metrics, achieving state-of-the-ar results (see Table 33 Appendix D) For instance, attained an FESR of 92.0%, surpassing the next-best model, Claude 3, by 4.6 points. While proprietary models like Claude and GPT-4o exhibit strong stability, they lag in critical areas such fuzzy detection and multi-turn consistency.
Entertainment, Autonomous Driving, and Comfort-to simulate diverse real-world user intents. The baseline model excelled across all scenarios, notably achieving 95.4% accuracy rate in safety-critical tasks, significantly outperforming competitors.
Additionally, we evaluated four specialized scenarios-Safety
Figure 5 presents each model's performance profile across four realistic vehicle-control six function-
scenarios Safety-critical, Entertainment, Autonomous Driving, and Comfort- call metrics. In every scenario, Qwen2.5-7B-33 (Ours) defines the Pareto frontier: its 95.4% Intent Recognition Accuracy in Safety-critical tasks outstrips GPT-4 by 7.4 pp: its 99.2% Counter-Question Question Coverage in Entertainment exceeds Qwen2.5-72B by 24.2 pp; in Autonomous Driving its 93.0% Dia- logue Consistency is 5.0 pp higher than the next best model; and its 97.5% Final Execution Success Rate in Comfort tasks is more than 8 pp above Claude 3. These gains demonstrate that compact LLM, when supervised with ClarifyVC-Data, can attain state-of-the-art robustness, coherence, and safety compliance in diverse, real-world driving interactions.
5 CONCLUSION
This work introduces ClarifyVC. unified framework that couples a schema- caneema-onsttrineed ambiguity- rich dataset (ClarifyVC-Data) with compliance-awane aware, three-tier evaluation protocol (ClarifyVC- Eval). By explicitly disentangling under-specification, clarification behavior, and long-horizon grounding, the protocol surfaces failure modes that single-turn accuracy obscures and yields diagnos- ties aligned with safety-critical deployment. The data pipeline preserves realism through real-world seeding and human validation, while enabling scalable synthesis of diverse ambiguity types. Em- pirically, Cl provides principled basis for comparing models and training strategies under uniform function-call semantics, addressing gaps in multi-turn clarification and protocol compliance measurement. Although experiments focus in-cabin voice control, the Clarify-Data framework is theoretically generalizable and can be adapted to other menectured-interattiin scenarios, such as interactive HCI and embodied agent systems, where reliable disambiguation and safe execution critical. Future work will integrate multimodal signals and release a public challenge to catalyze community benchmarking. We hope Clarify VC will serve as a foundation for rigorous research on ambiguity handling and robust interactive Al.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work is supported by Guangdong Basic and Applied Basic Research Foundation 2025A1515010300 Guangdong Province Project 1024QNIX00S Guangzhou Science and Technol- ogy Planning Project 2025A03J4491.
REFERENCES
Anthropic. Anthropic Releases Claude 3 AI Suite. hitpgs//ww..atthropoo..ccmm claude 3- familyl /y/ 2024.
Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. Benchmarking and improving text-to-SQL generation under ambiguity. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7053 7074, Singapore. December 2023. Association for Computational Linguisties. doi: 10.18653/112023.emmmg maa.emeeemnmmmmmn333663 main.436.
Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 31788-31812 Curran Associates, Inc., 2024.
Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi. Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. The art of saying no: Contextual noncompliance in language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Z.Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 49706-49748,
Curran Associates, Inc., 2024.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag. Edgar Dobriban, Nicolas Flammarion, George J. Pappas. Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In Globerson, Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Z.Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 55005-55029. Curran Associates, Inc., 2024.
Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadarshan, Anushka Jain, Arun K. Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, and K. Madhava Krishna. Talk2bev: Language-enhance bird's-eye view maps for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 16345-16352, 2024.
Changwoo Chun, Daniel Rim, and Juhee Park. Lim contextbridge: A hybrid approach for intent and dialogue understanding in ivsr. In Proceedings of the 31st International Conference Computational Linguistics: Industry Track, pp. 794-806 2025.
Juntao Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming Ji, and Yaodong Yang. Safesora: Towards safety alignment of text2video generation via a human preference dataset. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 17161-17214, Curran Associates, Inc., 2024.
Wenliang Dai, Samuel Cahyawijaya, Tiezheng Yu, Elham J. Barezi, Peng Xu, Cheuk Tung Yiu, Rita Frieske, Holy Lovenia, Genta Winata, Qifeng Chen, Xiaojuan Ma. Bertram Shi, and Pascale Fung. CI-AVSR: A Cantonese audio-visual speech datasetfor in-car command recognition. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Macgaard. Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 6796, Marseille, France, June 2022a. European Language Resources Association.
Wenliang Dai, Samuel Cahyawijaya, Tiezheng Yu, Elham J. Barezi, Peng Xu, Cheuk Tung Shadow Yiu, Rita Frieske, Holy Lovenia, Genta Indra Winata, Qifeng Chen, Xiaojuan Ma, Bertram E.

---

Published as a conference paper at ICLR 2026
Shi, and Pascale Fung. Ci-avsr: A cantonese audio-visual speech dataset for in-car command recognition. arXiv preprint arXiv:2201.03804, 2022b.
DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning,
2025. /ttpac/ees//aaxx....oo///////2.2
Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, and Marie Francine Moens. Talk2car: Taking control of your self-driving car. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2088-2095 2019.
Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, and Furong Huang. Easy2hard-bench Standardized difficulty labels for profiling Ilm performance and generalization. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances Neural Information Processing Systems, volume 37, pp. 44323-44366 Curran Associates, Inc., 2024.
Doubao Team. Doubao-1.5-prt ByteDance's Deep Thinking Model. bytedance. wwt.incccccccmooooooooeoeeeeoe zh/ http:///hh//cccaaaa/abaaaaa/e//a 1_r/_ 2025.
https: /ttps://seed
Albert Q. Jiang, Wenda Li, and Mateja Jamnik. Multi-language diversity benefits autoformaliza- tion. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 13600-83626 Curran Associates, Inc., 2024a.
Albert Q. Jiang, Alicja Ziarko, Bartosz Piotrowski, Wenda Li, Mateja Jamnik, and Piotr Mit os. Repurposing language models into embedding models: Finding the compute-optimal recipe. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.). Ad- vances in Neural Information Processing Systems, volume 37, pp. 61106-61137. Curran Associates, Inc., 2024b.
Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, and Percy Liang. Vhelm: A holistic evaluation of vision language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 140632-140666. Curran Associates, Inc., 2024.
Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. Lexeval: A comprehen- sive chinese legal benchmark for evaluating large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37. pp. 25061-25094 Curran Associates, Inc., 2024.
Kenkun Liu, Derong Jin, Ailing Zeng, Xiaoguang Han, and Lei Zhang- comprehensive benchmark for neural human radiance fields. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Advancess in Neural Information Processing Systems, volume 36, pp. 35107-35120. Curran Associates, Inc., 2023.
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng. Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. Apigen: Automated pipeline for generating verifiable and diverse function-calling
datasets. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 54463-54482. Curran Associates, Inc., 2024.
Yunsheng Ma. Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, et al. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15141-15151, 2024.
Kumar Manas and Adrian Paschke. Semantic role assisted natural language rule formalization for intelligent vehicle. In Anna Fensel, Ana Ozaki, Dumitru Roman, and Ahmet Soylu (eds.), Rules and Reasoning, pp. 175-189, Cham, 2023. Springer Nature Switzerland, ISBN 978-3-031-45072-3.
12

---

Published as a conference paper at ICLR 2026
Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo'llava: Your personalized language and vision assistant. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang ( Advances in Neural Information Processing Systems, volume 37, pp. 40913-40951. Curran Associates, Inc., 2024.
Eda Okur, Shachi H. Kumar, Saurav Sahay, Asli Arslan Esme, and Lama Nachman. Natural language interactions autonomous vehicles: Intent detection and slot filling from passenger utterances. In Alexander Gelbukh (ed.), Computational I Linguistics and Intelligent Text Processing, pp. 334-350, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-24340-00
OpenAI. GPT-4o: OpenAI's Multimodal Model.
hello- Boclo-gpt--0 40 2024a.
OpenAl. Introducing OpenAl ol: A New Reasoning Model Series. https://openal com/
2024b.
OpenAl et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2024.
Letian Peng and Jingbo Shang. Quantifying and optimizing global faithfulness in persona-driven role- playing. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances Neural Information Processing Systems, volume 37, pp. 27556-27583. Curran Associates, Inc., 2024.
Pradip Pramanick, Chayan Sarkar, Sayan Paul, Ruddra dev Roychoudhury, and Brojeshwar Bhowmick. Doro: Disambiguation of referred object for embodied agents. IEEE Robotics and Automation Letters, 7(4):10826-18833 2022. doi: 10.1109/LRA.222..195515
Parthib Roy, Srinivasa Perisetla, Shashank Shriram, Harsha Krishnaswamy, Aryan Keskar, and Ross Greer. doscenes: An autonomous driving dataset with natural language instruction for human interaction and vision-language navigation, 2024.
Irina Saparina and Mirella Lapata. Ambrosia: A benchmark for parsing ambiguous questions into database queries. In A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances Neural Information Processing Systems, volume 37, pp. 90600-90628. Curran Associates, Inc., 2024.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L. Waslander, Yu Liu, and Hongsheng
Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15120-15130, June 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Curran Associates,
Advances in Neural Information Processing Systems, volume 37, pp. Inc., 2024.
Pascal Sikorski, Kaleb Yu, Lucy Billadeau, Flavio Esposito, Hadi AliAkbarpour, and Madi Babaiasl. Improving robotic arms through natural language processing, computer vision, and edge computing. In 2025 3rd International Conference on Mechatronics. Control and Robotics (ICMCR), pp. 35 41, 2025. doi: 10.1 1LL111//CCC44990000222200 20..R//6899..22..100 10962987.
N. N. Sriram, Tirth Maniar, Jayaganesh Kalyanasundaram, Vineet Gandhi, Brojeshwar Bhowmick, and K Madhava Krishna. Talk to the vehicle: Language conditioned autonomous navigation of self driving cars. In 2019 IEEE/RS.J International Conference on Intelligent Robots and Systems (IROS), pp. 5284-5290, 2019.
Meta Llama Team. meta mete-laama/llaaa331--000---
https://huggingfaace
meta- lww.llaaaaaia Bamma/Llaaa-333
2024a.
Meta Llama Team. meta- matelllammlllmma-31---6-
https://uuginffaceeeu
metta- 1am-1lama/ii ama 3tt - 2024b.
13

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwen2.5: A party of foundation models. http://eenn...ithu....ob/b//o/ httpssiss.t//////oooooooaaaaaaad qwen2.5 5 September 2024c.
Megha Thukral, Sourish Gunesh Dhekane, Shruthi K. Hiremath, Harish Haresamudram, and Thomas Ploetz. Layout-agnostic human activity recognition in smart homes through textual descriptions of sensor triggers (tdost). Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 9(1), March 2025. doi: 10.1 14113712277. URL httphhtts///ee /ttps:://dii.o ozg. 1ttp///...11153122222
Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. The mamba in the Ilama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 2024a.
Yujin Wang. Zhaoyan Huang, Shiying Dong, Hongqing Chu, Xiang Yin, and Bingzhao Gao. Chatstl: A framework of translation from natural language to signal temporal logic specifications for autonomous vehicle navigation out of blocked scenarios. In 2024 16th International Conference on Computer and Automation Engineering (ICCAE), pp. 483-487, 2024b.
Yuxin Wen, Leo Marchyok, Sanghyun Hong. Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy backdoors: Enhancing membership inference through poisoning pre-trained models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 83374 83396. Curran Associates, Inc., 2024.
John Wenskovitch, Corey Fallon, Kate Miller, and Aritra Dasgupta. Characterizing interaction uncertainty in numan-maching teams. 2024 IEEE 4th International Conference on Human- Machine Systems (ICHMS), pp. 1-6, 2024.
Haoning Wu, Dongxu Li, Bei Chen, and Jumnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 28828-28857. Curran Associates, Inc., 2024.
Xiao Yang, Kai Sun, Hao Xin. Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang. Eting Yuan, Hanwen Zha, Nan Tang. Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen-tau Yih, and Xin Luna Dong, Crag - comprehensive rag benehmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. L.Tomzzak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 10470-10490. Curran Associates, Inc., 2024.
Jieyu Zhang, Weikai Huang. Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 19965-19974. Curran Associates, Inc., 2024a.
Qin Zhang, Sihan Cai, Jiaxu Zhao, Mykola Pechenizkiy, and Meng Fang. CHAmbi: new benchmark on Chinese ambiguity challenges for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 14883-1483 14898, Miami, Florida, USA. November 2024b. Association for Computational Linguistics. doi: 10. 18653/v 1/2024. htt.//.iindinnseeme..i.. findings-emnlp..""
Lianmin Zheng, Haichen Zhou, Liangsheng Yin, Haoze Wu, Tianyu Yue, Tianmin Shi, Yizhong Fan, Jeffrey Li, Zongheng Yang. Yiran Huang, Yuanshun Yao, John Langford, Ying Sheng, Harrison Chase, Tianqi Chen, and Xinyi Wang. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024.
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredríkson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.). Advances in Neural Information Processing Systems, volume 37, pp. 83345-83373. Curran Associates, Inc., 2024.
14

---

Published as a conference paper at ICLR 2026
A DATASETS
This section provides detailed information on the ClarifyVC-Data dataset, which underpins the evaluations presented in the main text. The dataset is designed support robust training and testing of large language models (LLMs) in vehicle control scenarios.
A.1 CLARIFYVC-DAY CONSTRUCTION
The ClarifyVC- dataset is constructed through a three-stage pipeline. Seed queries are derived from real-world user command corpora (collected from in-vehicle infotainment systems of major manufacturers during (022-2024) and synthetic functional specifications covering diverse vehicle control scenarios. The final dataset comprises 20,000 samples: 6,000 positive chains (unambiguous instructions), 8,000 negative chains (fuzzy, incomplete, or conflicting instructions), and 6,000 dialogue sequences (contextual interactions).
Types of Tasks Included in ClarifyVC-Datt ClarifyVC- is organized around the three capability tiers required by in-vehicle voice control systems. Each tier corresponds distinct task type(Table
These tasks directly reflect real production demands automotive HMIs, where user utterances vary widely from direct to vague to multi-step interactions.
To thoroughly evaluate in-vehicle voice control behavior, Clarify VC-Data includes diverse set of linguistic and functional variations. These variations reflect both the distribution observed in real in-car user logs and the ambiguity patterns that in-vehicle HMIs must handle. Below we summarize the major variation families (measured over the 20,436 ceed-aligned samples), ensuring balanced and realistic composition.
Table 3: Types of Tasks Included in ClarifyVC-DDe
Tier
Task Type
Description
Purpose
Tier
Single-turn structured parsing
Convert natural command into schema-aligned function call Identify underspecification and ask the correct clarifying question
Measures parsing & grounding
Tier 2
Ambiguity detection & clarification
Measures safety alignment & con- servativeness
Tier
Multi-turn dialogue grounding
Maintain context
Mcasures long-horizon consisteney & exccution precision
and recover missing parameters
Linguistic Variation Categories In Table Numbers sum 100% after rounding and reflect the natural ambiguity distribution in production logs.
Fuzzy Ambiguity Categories (Subset of Above, but Evaluated Separately) Because ambiguity central to our benchmark, also categorize utterances by fuzzy ambiguity type, which triggers Tier-2 evaluation in Table 5
These categories enable precise analysis of ambiguity handling behaviors.
Multi-Turn Dialogue Categories (MEA-Generated) For Tier-3 evaluation, Clarify VC includes multi-turn dialogues (generated via MEA), categorized by the type information missing from the initial command in Table
These multi-turn scenarios reflect actual conversational needs observed in deployed automotive voice assistants.
A.2 HYBRID BENCHIMARK CONSTRUCTION WITH
GROUNDING
We clarify that our benchmark is not purely simulation- based, but rather a hybrid approach that integrates extensive real-world grounding with controlled LLM-augmented generation. This de- sign ensures both scalability and authenticity, addressing concerns about human involvement and distributional overfitring simultaneously.
15

---

Published as a conference paper at ICLR 2026
Table 4: Linguistic Variation Categories.
Variation Category
Examples
Purpuse
Direct commands
"Set AC 22"
27%
Canonical structured parsing
Subjective modifiers
Fuzzy, intensity-hase based amhiguity
Omiited paramelers
"Tum on the lights."
22%
Missing slot recovery & clarification cues
Colloquial Iree-for form expressions
comty here."
Naturalness robusiness casual phrasing
Spatial references
"Open the left window."
9%
Eutity grounding & spatial disambiguation
Temporal pattems
"Turn it off for now?"
Handling temporal modifiers
Multi goal composite commands
"Warm the front and cool the hack,"
4%
Conffict detection & malti entity constraints
Implicit comparisons
"Make the hack cooler than the front."
Relative parameter resolution
cenlext-dependent follow ups
"Da the same as yesterday."
3%
Long term contextual inference
Table 5: Fuzzy Ambiguity Categories.
Ambiguity Type
Example
% among Fuzzy Samples
Purpose
Intensity ambiguity
"A bit warmer/ / slightly brighter"
41%
Requires clarifying magnitude
Boundary ambiguity
"Open it halfway / just little"
24
Requires mapping vague intervals
Entity ambiguity
"Open (which window?) ?)
19%
Requires clarifying the referent
Mode ambiguity
"Make comfortable"
10%
Must ground specific HVACkeat mode
Referential ambiguity
"Tum that off""
6%
Requires anaphora resolution
Real-World Data Foundation and Human Validation The foundation of our dataset consists of over 20,000 carefully selected real-world user utterances from an extensive, industrial corpus of 4 million authentic vehicle interaction logs datasets are rarely accessible in academic or industry research, highlighting the benchmark's 's unique value.
Such large-scale automotive voice
After generation, dialogues underwent rigorous real-world validation. Domain experts manually validated dialogues through practical in-car system simulations, ensuring alignment with actual usage scenarios and filtering out unnatural artifacts. This human-in-the-loop validation included:
Manual verification of dialogue coherence and realistic ambiguity representation. Hands-on testing in real-car infotainment system simulations. Retention only of dialogues meeting strict human validation criteria.
Additionally, we performed a dedicated human human-grounding validation test on a sampled subset of 500 generated dialogues. The results (Table demonstrate high scores across linguistic realism, ambiguity plausibility, dialogue coherence, and practical applicability, confirming our benchmark's 's real-world relevance.
Characterization of the Real In-Vehiele Logs The real-world logs come from 4M+ production voice interactions collected across multiple vehicle models, cabin configurations, and firmware versions. Below we provide additional statistics to characterize their distribution and linguistic properties. From the raw logs, the schema-filtered 20,436 usable samples cover all major in-cabin domains (Table 9). This distribution closely matches that observed in commercial vehicle HMI deployments. For privacy reasons, raw OEM logs are never released; moreover, Clarify VCtaata does not contain any raw logs, as all commands undergo intent extraction, parameter normalization, entity canonicalization, and removal of unsafe or identifying content before schema instantiation and variant generation.
To clarify linguistic properties, we computed length and structural statistics, as summarized in Table 10 These statistics highlight why ambiguity handling is a critical component of the task.
Real logs contain many naturally ambiguous or risky forms:
21% % involve incomplete thermal commands.
8% reference windows/sunroof without direction.
5% issue potentially unsafe actions ("open it" while driving).
3% require multi-turn grounding.
Such patterns motivated the design of Tier 2 and Tier 3 evaluation criteria avoid unsafe behaviors.
16

---

Published as a conference paper at ICLR 2026
Table Multi-Tum Dialogue Categories.
Multi-Turn Dialogue Type
Description
Example
in Multi-Turn
Parameter completion
Missing semperature brightaces / dagree
"Make it warm." - "What cemperature.""
46%
Entity clarification
Missing window/seat/zon reference
"Cool it." - "Which window?"
28%
Conffict resolution
Conflicting constrinss
keep vents "Set to comfort mode." - "Which preset?"
Preference grounding
Mode or preset selection
9%
Temporal sequencing
Multi-step actions
"hum off after moment."
Table 7: Comparative analysis of vehicle command benchmarks.
Benchmark
Ambiguity Handling
Multi-
Safety
Evaluation
Distil-
turn
Constr.
Protocol
lation
Talk2Car Deruyttere et al. 2019
Basic
No
Partial
Single- round
No
APIGen (Liu al. 2024)
Synthetic
Single- round
Yes
Easy2Hard DDng al. 2024
Difficulty lev-
No
No
Single-
No
els
round
CI-AVSR (Dai
2022a
Moderate Visual only
Limited
Two-tier
No
doScenes Roy et al. 2024
No
No
Single- No round
Clarify VCtaata
9 types
Yes
Full
Three-tier
Yes
Controlled Generation and Distributional Alignment To prevent feedback loops and overfitting,
employ a strict separation between generation and evaluation models:
Generation: DeepSeekR 1-based instruction-tumed LLMs.
Evaluation: GPT-4, Qwen2.5, Claude 3 (architecturally distinet from generation models).
We further ensure realism and consistency through structured prompt engineering at each agent stage:
SPA prompts enforce inter-entity schemas and intent-slot alignment.
FLA prompts introduce ambiguity via realistic linguistic perturbations.
AGA prompts inject adversarial ambiguity under entropy thresholds.
MEA prompts maintain multi-turn coherence and causal continuity.
To quantitatively assess distributional alignment, we compared synthetic and real dialogues across multiple in-cabin scenarios. As shown in Table intent coverage and KL-divergence values confirm close fidelity to woaldwrld usage distributions.
This hybrid
wwweodooooyyoombbiie real-world seeds, human validation, and controlled LLM generation-ensuree that our benchmark is both scalable and faithful to real-world automotive voice interactions. We will explicitly highlight these aspects in the revised manuscript to reinforce the benchmark's ' credibility and practical relevance.
The design of our Clarify Vv-lval: objectives, challenges, and metries
The specifics of this
design are summarized in Table I2
A.3 AMBIGUITY INJECTION RELIABILITY
Before designing our pipeline, we conducted an initial assessment two baseline LLM-bbaee rewriting strategies. Specifically, evaluated GPT-4-Turbo prompted to "add fuzziness" and Qwen2.5-32B prompted to "rewrite with ambiguity." Using 500 real user commands, we compared intent labels before and after rewriting. Results shown in Table [3
These findings indicate that naive LLM rewriting often overshoots, leading to paraphrasing, impliciti- zation, unintended semantic urifr-ludes for controlled fuzz generation.
17

---

Published as a conference paper at ICLR 2026
Table Blind human comparison of dataset realism and usability. Annotators rate 500 shuffled dialogues per dataset (Talk2Car, doScenes, CI-AVSR, Clarify CaarifyCC-Deie VC-Data) on a 1-5 Likert scale. Higher better. VC-rifyVC-DDi achieves the highest scores across all aspects.
Dataset
Ling. Realism
Ambig. Plaus.
Dial. Coh.
Prac. Applic.
Talk2Car
4.2 0.3
3.9 0.4
4.0 + 0.3
4.1 0.3
doScenes
4.1 + 0.3
3.8 1.8+0.4
3.9 + 0.3
4.0 0.3
CI-AVSR
4.0 0.4
3.7
3.8 0.4
3.9 0.4
ClarifyVC-Data
4.6
4.6 0.1
4.7 1 0.2
4.5
Protcol. 5 annotators, blind and shuffled setting: each row uses the same rating interface us Table b). Inter annotator agreement (Fleiss' rc) overall.
Table 9: Distribution of User Command Domains
Domain
% in Logs
Example Commands
HVAC/ Thermal Comfort
34%
"It's too hot hot-cool it down." "
Lighting
18%
"Turn on ambient lights."
Windows Sunroof
15%
"Open the rear-right window."
Seat / Comfort
12%
"Heat my seat."
Navigation
10%
"Take me to the nearest station."
Media / Infotainment
11%
"Play my jazz playlist."
To further quantify this mismatch, computed intent KL-divergence between the 20,436 original seed commands and the naively rewritten FIA samples, obtaining 0.241 at the token level and 0.3 12 at the intent level. Such divergence is sufficiently large alter downstream model behavior, validating the reviewer's s concern about uncontrolled fuzz distortion.
To address the semantic drift observed in unconstrained LLM rewriting, Clarify adopts drift- control design that combines model specialization, targeted supervision, and strict prompting con- straints. SPA, FIA. and MEA rely on DeepSeek-RI, which shows stable behavior under constrained rewriting. while AGA uses Qwen2.5-72B with prosriel-restrieted decoding and JSON-defined intent boundaries. Each component is paired with the model whose inductive bias aligns with its specific requirements.
second layer of control is provided through supervised fine-tuning on 2,000 expert-curated seed-fuzzy pairs, created by three in-vehicle HCI specialists with verified intent consistency. This data allows the model to learn controlled ambiguity- as softened intensity or reduced speci- fichty-without altering the underlying action, entity, goal.
The efficacy of this method is demonstrated Table 14 which shows the quantitative reduction in semantic drift. The Intent KL divergence was reduced from 0.3 to negligible 0.062, and the intent preservation rate was raised to 98, 9%. This confirms the high semantic stability and distributional faithfulness of the augmented dataset.
B EXPERIMENTAL. SETUP
This section details the experimental environment and the agent-orchestrated pipeline used to generate the evaluation test sets, as referenced in Section of the main text.
B. SOFTWARE AND HARDWARE ENVIRONMENT
Experiments were conducted using Python 3.10 on servers equipped with Intel Xeon Platinum 8380 CPUs and NVIDIA A100 GPUs, running Linux operating system. This configuration ensures efficient data processing and model training for the agent-orchestrated collaborative generation framework.
18

---

Published as a conference paper at ICLR 2026
Table 10: Utterance Length and Structure Statistics.
Statistic
Value
Mean tokens per utterance
7.8
Median tokens
6
90th percentile
14
% containing subjective modifiers ("slightly", "a bit")
22%
containing explicit parameters
% containing explicit parameters
39%
Table 11: Distribution alignment between synthetic and real-world dialogues.
Scenario HVAC Control
Intent Coverage (Real)
Intent Coverage (Synthetic)
KL-DDvergence
93.8%
95.2%
0.06
Infotainment System
89.2%
90.0%
0.05
Navigation Commands
91.5%
90.9%
0.04
Comfort Adjustments
88.7%
89.1%
0.05
B.2
AGENT-ORCHESTRATED GENERATION PIPELINE
The test sets were generated using an agent-orchestrated pipeline comprising four agents, each responsible for specific task in creating complex, ambiguous, and multi-tury commands.
Semantic Parsing Agent (SPA). The SPA parses an input command c into a semantic representation E, P), where I, E, and P denote intent, entity, and parameters, respectively. The prompt is:
Promptspa = "Given the command: intent, entities, and parameters in the format (I, E, P)."
(c), extr the
The semantic consistency score SC validates the prompt's ' effectiveness:
SC =
Σ T(i=i))
(6)
SE(LE,P)
prompt considered valid SC > 0.9 on validation set.
Fuzz Injection Agent (FIA), The FIA introduces ambiguity into commands, generating a fuzzed version c'. The prompt is:
Promptria = "Given the command: (c), introduce ambiquity by (E) with intensity (e), where {ʃ} is one of {omil parameler, subjeclive expressionn...."""
Ambiguity types are sampled from categorical distribution, optimized to maximize entropy EreF dy log oy for diverse coverage.
entropp H(F)
Multi-Turn Evolution Agent (MEA). The MEA generates dialogue sequences D
((c,,)) (c), The prompt is:
Promptwma "Given the dialogue history: (c), rı)
b, generate the next system response and user command 6t11."
Dialogue coherence is measured by:
T-1 DC
Σ
cos(hr,he41)
(7)
T
19

---

Published as a conference paper at ICLR 2026
Table 12: CC-rilyVC-Eval: objectives, challenges, and metrics.
Tier
Objective
Challenges Captured
Metries
Tier Sougle-Round Fuzzy Parsing
Purse mildly ambiguous single- turn commands.
parameters, vague references. subjective expressions.
Under-specified
IRA: ΣN 104, ())
PEP:
PCP
1/1
Tier 2: Extreme Fuzzy Counter-Questoonnng
Detect severe ambiguity and ask clarifying questions.
Extreme uncertainty. vague
FDR: IPH
TP
nouns, clarification relevance.
CQC: Human-rated (1-
5)
Tier 3: Dynamic Multi- Turn Understanding
Retain context and execute accu- mulated commands.
Multi-turn dependency, memory. paramcter coherence.
DC:
E 008(3),8411
FESR:
Σ> [(xec(())
(x)c(c
Table 13: Intent Preservation and Drift Analysis Naive LLM-Based Rewriting Methods
Model
Intent Preservation Rate
Drift Examples
GPT-4-Turbo
92.1 %
"Open the left window" _
"I "Let some fresh air" (intent shift: window - HVAC/ventilation)
Qwen2.5-32B
89.4%
"Turn off reading lights" - "Make it darker in here" (intent compression: explicit -> implicit)
prompt effective DC 0.85.
Adversarial Generation Agent (AGA). The AGA generates adversarial commands Cadv using protocol-constrained seed queries. The prompt is:
Promptxca "Given the command: (c), refer to the slot information provided in the protocol constraints, generalize the seed query, and generate an adversarial variant by introducing extreme ambiguity while keeping it plausible."
Adversarial strength AS is measured as the perplexity of Cadv. computed using pretrained GPT-2 model.
The joint probability of generating complete sample via the full pipeline is expressed as:
P(caveee) = P(slc, PromptsPA) P(Cavvls, Prompt P(cllc) = P(carlc) P(d Promptpra)
(8)
(9)
P(D(e) = P(c'cc) P(D]e', Prompt
(10)
PutalCCuty c' D ()c) = P(caar/c) + +
(11)
Pipeline Algorithm.
The agents integrated into pipeline, shown in Algorithm |
Example of Stage Effeets A concrete running exabple(Table 15) clearly illustrates how a real in-vehicle utterance evolves through the pipeline.
Agent-orchestrated Generation Framework Components tuned open-source LLMs via prompt orchestration, without fine-tuning, as detailed in Table [6
The generation agents use instruction-
These models were chosen based empirical performance during pilot generation trials across ambiguity types and domains.
Crucially, orchettrche framework is model-agnostic: each agent relies standardized prompts and schema constraints, enabling drop-in replacement with other models (c.g.. GPT-4, Claude ChatGL LLaMA3) without modifying the overall pipeline logic.
20

---

Published as a conference paper at ICLR 2026
Table 14; Ambiguity Injection Reliability
Stage
Token KL
Intent KL
Intent Preservation
Naive FIA (baseline) ClarifyVC FIA (ours)
0.241 0.047
0.312 0.062
89-92%
98.9%
Table 15: Stages and Representations
Stage
Representation
Example (informal)
Change
Seed (real log)
User utterance (1. E, P)
"It's too hot, adjust the air conditioning." Tntont-SeTemmerrturr..
Raw in-car request Structured grounding
SPA
cntit==Frontabin,
temp=22"C
AGA
Adversurial command
"Set the front cabin 22 and keep the unchanged."
Adds constraints/confliets
FIA
Fuzzy command Multi-turn dialnguee
"Make the front bit cooler than it is now."
Injects ambiguity Recovers parameters
MEA
Q: "What exact temperature?"" "22 degrees."
Table 16: Agcnttorchestrrated generation framework components
Agent
Implementation
Primary Function
SPA.FIA and MEA
DeepSeek-Ve -L-RRI
Semantic parsing, fuzzing, and multi-turn evolution
(API)
AGA
Qwen2.5-72B
Protocol-conssrraiee adversarial in- struction generation
(vLLM)
Algorithmic Novelty and Ablation
We provide ablation experiments analyzing the importance of
agent ordering in Table 17 and Table 18
Table 17: Ablation study of agent ordering and composition on key performance metrics
Pipeline Variant
Ambiguity Diversity (1)
Dialogue Coherence (1)
Protocol Adherence (1)
SPA + AGA - FIA - MEA (default)
0.85
0.88
0.98
SPA FIA AGA MEA
0.79
0.82
0.95
AGA SPA FIA MEA
0.75
0.78
0.92
SPA + AGA - MEA (without FIA)
0.68
0.84
0.94
SPA + FIA - MEA (without AGA)
0.71
0.85
0.95
MEA (without FIA AGA)
0.55
0.87
0.96
Key Observations:
The default pipeline PPA-AAA-FFAA-AA achieves optimal balance among diversity, coher- ence, and adherence, confirming intentional design.
Algorithm 1 Agents Pipeline Algorithm
Require: User Command c
Ensure: Data Pools (Tier-1, 1, Tier-2, Tier-3)
1: procedure PROCESSCOMMAND(G)
2: + SPA(c)
>> Semantic Parsing Agent Adversarial Generation Agent
3: Cadv + AGA(ss)
4: DataPoolfer. DataPool U [Catc)
5:
d' FIA(Cadv)
Fuzz Injection Agent
6:
DataPooltier'
+ DataPoolnera U [c')
7:
D MEA(c')
Multi-turn Evolution Agent,
8:
D D= (ei,rii (C., (n))
9:
DataPoolyer- +- DataPool U D
10: end procedure
21

---

Published as a conference paper at ICLR 2026
Table 18: Ablation Study: Causal Contribution of Individual ClarifyVC Pipeline Components
Removed Component
Avg Score Drop
Interpretation
-SPA (semantic parsing)
-9,4%
Loss of schema aligmment + causal contributor
strong
-AG (adversarial generation)
-6.7%
OOD coverage weakened
-IAA (fuzzy injection)
-11.1%
Robust uubiguity-handling collapses =
largest causal effect
-MEA (multi-tum evolution)
-4.6%
Context-grounding weakened
Uaalegue Cobyrefice
all
Ambiguity Civersity
Protocol Adhieence
Figure 6: Performance Comparison of Pipeline Variants
Switching FIA and AGA reduces diversity, indicating FIA's fuzz injection effectiveness decreases without structured adversarial perturbation first.
Removing either AGA FIA notably decreases ambiguity diversity, underscoring each agent's essential role.
Omitting both agents severely reduces ambiguity, though coherence remains high, emphasizing the necessity of the pipeline for balanced ambiguity.
Variability statistics Variability statisties and significance tests are important demonstrate the robustness and reproducibility of our benchmark results. To explicitly address this concern, conducted additional experiments by performing independent evaluation runs for each primary model configuration reported in Table 33 We now include detailed mean standard deviation values along with significance testing in Table 19 paired t-tests vs. GPT-4 baseline).
Table 19: Model Performance Stability and Significance Tests
Model
Protocol Adherence (%)
IRA (%)
Dialogue Coherence (%)
p-value (vs GPT-4)
GPT-4
98.2 + 0.2
94.0 + 0.3
89.5 0.4
Qwen2.5-7B
97.8 0.2
91.2 i
86.7 0.5
0.01
Claude
96.3
90.8 1 0.4
86.0 0.6
0.01
LLAMA3-8B
95.5 + 0.3
88.2 +
83.4 0.6
< 0.01
22

---

Published as a conference paper at ICLR 2026
Observations:
Standard deviations < 1%, confirming stability across independent runs.
Paired t-tests indicate statistically significant differences, validating discriminative power.
Additionally, our Data Generation Phase is deterministic due structured prompting and controlled API constraints, making multiple runs unnecessary at this stage.
B.3 DETAILED SPECIFICATION OF EVALUATION DATA
Training Data Summary Our training corpus originates from 4M+ anonymized in ne-veiel voice logs collected from production vehicles. We construct a high-quality, schema-aligned training set through multi-stage filtering, parsing, safety checking, and protocol normalization. The final training set used for ClarifyVC- VC-Models is illustrated in Table 20
Table 20: Training Data Summary.
Source
Count
Description
Raw production logs
4,000,000+
Real user voice interactions covering diverse operations Passing intent extraction, parameter validation, schema grounding. safety constraints We deliberately train only on Clarify VC-Data avoid bias Used for SFT (e.g.. Qwen2.5-7B-SF
Schemna Schema-filerre seed commands
15,000
VCarityC-Datt
20.436
Final training size
==20k samples
Benchmarks used in 'Evaluation on Advanced Scenarios" The "Advanced Scenarios" evaluation consists of three tnttoo-dditrbbutionOOO
test sets, each containing 5,000 samples (total: 15,000). These test sets are fully disjoint from all training data, including Clarify VC-Data, raw logs, and any augmentation outputs(Table 2T.A three benchmarks seed utterances that do not appear in training and do not share augmentation lineage with the CCarifyVC-Da used for SFT. This ensures strict train-test separation.
Table 21: Benchmarks Used in Evaluation Advanced Scenarios
Test Set
Size
Description
Data Source OOD Justification
(I) Single-round fuzzy parsing
5,000
Communds containing intensity fuzziness, boundary amhiguity, im- plicit parameters
Generated via FIA, using non- overlapping seed commands (held- out from the 20k Clarify VC-Data) Generated from distinct real included in Clarify VC seeds: ambi- guity amplified AGA+FI
(2) Extreme- counter--ueettonnnng
5,000
Highly ambiguous commands quiring clarifying questions
(3) Multi-turn evolving-context dialogues
5,000 2-4 tum interactions with shifling referents, evolving context, and pa-
Produced via MEA using held-out interaction templates and indepen- dent schema parameter ranges
rameter recovery
The advanced-scenart benchmarks out-of-distribution (OOD) relative to the training data due to distinct seed sources and measurable distributional divergence. The 15,000 test samples are derived parameter spaces, none of which
from held-out raw logs, unseen templates, and overlap with the 20,436 seeds used for training. Furthermore, as summarized in Table [2] token-level and intent-level KL divergence between the Clarify VC training set and each advanced benchmark confirms non-trivial distribution shift. These results substantiate that the evaluation is not only data-disjoint but deliberately designed to assess OOD generalization.
Table 22: KL Divergence Between Clarify -r-nriin and Evaluation Sets
Comparison
Token KL
Intent KL
VC-ifay Advanced-Fuzzy 0.073
0.141
Clarify VC-Train + Extreme-Fuzz
0.082
0.132
ClarifyVC Multi-Turn
0.145
0.197
The evaluation also assesses performance across four defined application scenarios scenarios-SSffty-criiee Safety-critical, Autonomous Driving, Entertainment, and Comfort- are not newly constructed datasets, but
23

---

Published as a conference paper at ICLR 2026
scenario-conditoone subsets derived from the three OOD test sets via re-labeling with scenario- specific tags. As illustrated in Table 23 each scenario corresponds to specific data sources and exhibits distinct OOD characteristics. Model results demonstrate that our Clarify -Cuseery-ee 7B model maintains Pareto-optimal perfoommmcc performance across all four scenarios.
Table 23: Scenario-wise OOD Evaluation Settings
Scenario
Data Source
Example
OOD Justification
Safety criticl
Extreme: Fuzz Multi Turn
"Open it bit ... (which window?)"
Held out logs + unseen fizzz patterns
Autonomous Driving
Multi-Tune only
"Keep lane assist unless
Comes from unseen templates
Entertainment
Fuzzy Parsing
"Play something relaxing ... (genre?)"
Parameter space randomized
Comfort
All three
HVAC seating multi-gon goal requests
Different entity-slot frequencies
C EVALUATION PROTOCOLS
This section describes the three-tier introduced in the main text (Section 4).
evaluation framework used to assess model performance, as
Table 24: The three-tier ClarifyVC-Fval VC-Eval protocol to evaluation metrics. Each tier isolates a distinct family of failure modes while jointly covering the spectrum of function-call understanding and safe execution.
Tier
Metrics Used
Rutionale
Tier 1: Single-Round Instruction Fuzzy ing
Round
Intent Recognition Accuracy (IRA), Pu-
Captures the model's ability to resolve under specified single tur com- intents
rameter Extraction Precision (PEP), Intent Hit Rate (IHR), Function Hit Rate (HHK)
calls, These metrics reflect nccumey and parameter precision the most basic function-call level.
Tier 2: Extreme Fuzzy
Puzzy Detection Rate (FDR), Counter-
Exaluntes whether the model istentifies extreme ambiguity and adopts safe clarification strategies insteud of unsafe guesses. Metrics track conservative ve behavior, protocol adherence, and safety awareness.
Instruction Counter-
Question Coverage (CQC), Protocol Com- pliance Rate (PCR)
Questioning
Tier 3: Multi-tura Dia- logue
Dialogue Consistcacy (DC), Final Eseeu- tice Success Rate (HESR), Parameter Com- pleteness (Fl-score)
Assesses ongghorizzon intoractions where the model must gather missing semantics over multiple turns, maintain coherence, and ultimately ground executable commands. metrics mensure culmination dialogue fidelity and execution success.
Table 25: Definitions of evaluation metrics used in CI. All metries are defined in this work to capture different aspects of fuzzy command understanding and function-call execution.
Metric
Definition and Description
Intent Recognition Accuracy (IRA)
Measures whether the model correctly identifies the target intent (egg. HVAC adjustment, navigation command) from fuzzy underspecified natural langunge instraction. Equivalent semantic classification accuracy at the intent level. Evaluates correctness of parameter extraction (c.g.. temperature value, media type, destination) given an identified intent. Precision is computed against gold-standard annotations to ensure valid executable function calls.
Parameter Extraction Predi- sion (PEP)
Fuzzy Detection Rate (FDR)
Captures the propection of ambiguous underspecified instructions where the model successfully detects the presence fuzziness or uncertainly instead of over confidently execuling an unsafe action. High FDR reflects sufety aware hehavior.
Counter-Question Coverage (CQC)
Quantifies how often the model responds with clarification questions cases of ambiguity, rather than hallucinating parameters guessing. Coverage measured as the ratio appropriste countcr-questioss to total ambiguous instructions.
Dialogue Consistency (DC)
Assesses model's ability to maintain semantic referential coherence across multiple tums of clarifica- tion. Consistency is measured tracking dislogue alignment and the absence of contradictions. Measures whether the final resolved command (after possible clarifications) leads to a sufe and correct function execution the system. This combines suceessful intent detection, parameter extraction, and anbiguity resolution.
Final Execution Success Rate (FESR)
Intent Rate (1HR)
Evalustes whether the predicted intent label exactly matches the gold-standard intent. This focuses purely on intent level accuracy independent of parameter filling.
Function (FHR)
Checks whether the predicted API/function name aligns with the gold-snn standard function call. ensures the correct system API invoked.
Parameter Completeness (FI- Score)
Measures hoth the precision and recall extracted slots/parameters within the predicted function call. Fl balances coverage of required arguments with correctness extracted values.
Protocol Compliance Rate (PCR)
Assesses whether generated function calls comply with predefined API schema and safety constraints (c.g.. correct types, no missing required arguments, no unsafe defaults). High rellects reliability deployment.
24

---

Published as a conference paper at ICLR 2026
C.I TIER 1: SINGL-ROOUD FUZZY PARSING
This tier evaluates the model's ability to interpret ambiguous single-turn commands with subtle ambiguities (e.g., "Increase the temperature" without target value). Metries include:
Intent Recognition Accuracy (IRA):
N 144 4))
IRA
(12)
N
Parameter Extraction Precision (PEP):
[P n P JP]
PEP
(13)
C.2 TIER EXTREME FUZZY COUNTER-QUESTIONES
This tier tests the model's ability to detect and clarify highly ambiguous commands (e.g., "Turn that switch )f Metries include:
Fuzzy Detection Rate (FDR):
TP
FDR
(14)
TP FN
Counter-Question Coverage (CQC):
CQC Σ min(m,, M,) Σ Mj
(15)
C.3 TIER 3: DYNAMIC MULTI-TURN COMMAND UNDERSTANDING
This tier evaluates context retention and command execution in multi-turn dialogues. Metries include:
Dialogue Consistency (DC):
T cos (5. 8(41)
DC
(16)
T
Final Execution Success Rate (FESR):
N
FESR
((x Exec(c))
(17)
C.4 SUPPLEMENTARY INSTRUCTION
Sensitivity of realism threshold (inverse perplexity) We performed a sensitivity analysis on inverse perplexity (IP) realism thresholds across various percentiles in Table 26
Table 26:
Performance metrics under different inverse perplexity thresholds
Threshold (IP)
Ambiguity f Protocol f
Realism t
DQS f
5th (strict)
0.83
0.95
0.81
0.860
10th
0.86
0.94
0.81
0.869
20th (used)
0.89
0.95
0.82
0.887
50th (loose)
0.90
0.92
0.75
0.861
Striet filtering reduces diversity; loose thresholds reduce realism.
The chosen 20th percentile optimally balances realism, diversity, and adherence.
25

---

Published as a conference paper at ICLR 2026
Protocol
5th (strict)
10th
... 50th (loose)
0.95
0.90
0.75
Realism
70
Ambiguily
DQS
Figure 7: Performance Metrics under Different IP Thresholds
Rationale for Å₁ A = (0.4, 0.3) in Eq. (2) The weight combination Å - = 0..4, 0.3) used in Eq. (2) selected based careful expert consideration of the domain-specific importance of each evaluation dimension:
Ambiguity Diversity (AD=0.4): Primary goal to capture diverse ambiguities.
Protocol Compliance & Realism (PC/R=0.3 each): Essential for validity and authenticity.
Robustness analysis across alternative weights confirms minimal variation in aggregate scores and stable rankings, reinforcing the chosen default (0.4, 0.3, 0.3) configuration in Table 27
Table 27: Robustness analysis of weight configurations
Weights (AD, PC,
f
PC
DQS(Aggregate Score)
Rank Change in Table
(0.40, 0,30, 0.30) Default
0.89
0.95
0.82
0.887
(0.33, 0.33, 0.34) Balanced
0.89
0.95
0.82
0.886
No change change change change
(0.50, 0,25, 0.25) Emphasis on AD
0.89
0.95
0.82
0.886
(0.30, 0.50, 0.20) Emphasis on
0,906
(0.20, 0.20, 0.60) Emphasis
0.89
0.95
0.82
0.860
Gold-Standard Function-Call Definition. For all experiments in Section 4 especially the Function Rate evaluation in Section 4.3, the gold-standard structured function call of for each instruction is determined by unified validation pipeline tailored to production in-vehicle HMI constraints. A model prediction è; counted as correct only if it satisfies all criteria below.
Gold standard 1 1-1 Semantic Ground-Treee Truth Alignment. Each instruction is paired with an expert-annotated canonical function call under our unified schema (37 actions, 146 parameters). A prediction must match this call exactly in: (i) action name, (ii) required slot set, (iii) parameter values and types, and (iv) absence ballucinated arguments.
Gold standard 2 -Schemaaaaaaa and Type Validation. Predicted JSON is checked by deterministic schema validator that enforces: (i) action-slor legality, (ii) data-type correctness, (iii value-range constraints (e.g., brightness& [0, 100)), and (iv) mutual-exclusion nules between parameters.
26

---

Published as a conference paper at ICLR 2026
Gold standard Protocol and Safety Compliance. Beyond semantic matching, predictions must satisfy production HMI protocol rules, including: (i) deteeded constraints (e.g., tempera- ture changes allowed only when HVAC is ON), (ii) safety-dependent conditions (e.g., seat heating only if occupancy is detected), (iii) forbidden action-parameter combinations, and (iv) restrictions on overriding locked states (e.g., child-lock windows).
Gold standard 4 --ultti-ttep Ordering and Causal Preconditions. Some instructions trigger multi-step calls (e.g.. "navigate home and play music"). Gold-standard sequences additionally satisfy:
Functional order compliance: e.g., start navigation -> then play music (not reversed).
Causal preconditions: e-g.. defogging enabled before adjusting windshield airflow; AC mode compatible with semperature-setingg operations.
prediction violating ordering or causal rules is marked incorrect even individual calls are syntactically valid.
Gold standard 5 Real-Log Consistency (for Log-Derived Samples), For instructions originat- ing from real in-vehicle logs, we additionally verify that predicted calls are consistent with historical execution patterns by retrieving similar utterances from 4M+ anonymized logs via embedding search. Similarity thresholds and mappings were reviewed by two OEM experts to ensure real- wolldwrlld alignment.
Gold standard Human Adjudication for Ambiguous Cases. For inherently ambiguous or multi-intent instructions (~11%% of the evaluation set), predictions from all models are anonymized, shuffled, and adjudicated by human annotators under a blind protocol. Only predictions consistent with the majority vote are accepted as hits.
Gold standard 7 are satisfied:
Final Decision Rule. A prediction counts as correct only if criteria above
N
FHR
N Σ
[Alicrtteriasaciofeee(((
Table 28 summarizes the validation pipeline.
Table 28: Gold-standard validation criteria for function-call evaluation.
Criterion
Validator
Applies To
Semantic correctness
Expert annotations
All samples
Schema type validity
JSON schema validator
All samples
Protucol & safety compliance
HMI rule engine
samples
Multi-step order & causality
UX/causal rules
Multi-step calls
Real-log consistency
AM-log similarity check
Real-log log samples
Human majority vote
Blind adjudication
Ambiguous samples
Causal relationships analysis in model performance Cross-dataset generalization results for both backbones are presented in Table 29 demonstrating the superior performance of Clarify VC-Data across all evaluation settings.
Across two backbones and three evaluation suites, Clarify VC-trained models outperform all alterna- tives by 16-23%. These gains persist across entirely different evaluation distributions and are not tied to log-derived data. They arise from the structure and quality of Clarify rather than model size, training-set set scale, or domain overlap. Consequently, the dataset's 's design is the causal driver of the improvements.
Through controlled ablation, cross-dataset validation, and mechanistic evidence from Table 2 and Figure 3 we show that the observed performance gains are not attributable distributional overlap, model scale, data quantity, or log-derived exposure bias. Instead, they are causally driven by Clarify schema- grounded parsing (SPA), adversarial diversity (AGA), ambiguity- focused fuzz injection (FIA), multi-turn recovery supervision (MEA), and the overall difficulty profile of Clarify Data.
27

---

Published as a conference paper at ICLR 2026
Table 29; External Causal Validation of CaarifyC--aaii Model Performance Trained on Various Datasets and Evaluated on Unseen, Unbiased Benchmarks
(a) Qwen2.5-7BBIIsttmn Fine- Tine-Tuning Results
Training Set
Unbiased Test (4k)
Real Logs (2k)
OOD Fuzzy (1k)
Avg
ClarifyVC-Data (20k)
92.7% %
94,1 %
89.5%
92.1 %
NL2API-Car
72.4%
74.8%
67.1%
71.4%
Glaive-fe-v2
68.3%
71.0%
61.5%
66.9%
Talk2Car-FC
70.2%
73.6%
65.7%
69.8%
doScenes-FC
67.5%
70.4%
62.9%
66.9%
Mixed-Open
75.1%
76.3%
69.4%
73.6%
(b)
LLaMA3-8B-Instuulll Fine-Tuning Results
Training Set
Unbiased Test (4k)
Real Logs (2k)
OOD Fuzzy (1k)
Avg
ClarifyVC-Data (20k)
87.9
90.5%
85.1
87.8%
NL2API-Car
63.4%
66.1% %
58.2%
62.5%
Glaive-fe-v2
60.8%
63.2%
54.7%
59.6%
Talk2Car-FC
64.9%
67.5%
59.4%
63.9%
doScenes-FC
61.1%
63.8%
56.0%
60.3%
Mixed-Open
66.7%
69.0%
61.5%
65.7%
To further validate that these gains are causally tied to the pipeline-level design of (CrrfyyC-Datt rather than any vehicle-control prior-we additionally apply the full SPA/AGA/FIA/MEA pipeline to three non-automotive function-calling datasets: function-calling-chee chattl (general assistant APIs for finance, polling, and utilities), Glaive-fc-12 (open- task assistant for books, tools, and utilities), and Arabic Function Calling (multilingual scientific and utility API calls, e.g., boiling/mmeltin points). These corpora differ from in-cabin voice control in schema format, interaction pattern, domain, and linguistic style, and contain no car-related APIs.
In each case, we treat the original corpus as a "base" dataset and use Clarify SPA/AGA/FIAMEA pipeline purely data augmentation mechanism. Importantly, the pipeline only requires: structured JSON schema, (ii) typed parameters, and (iii) optional safety constraints; it does not encode any automotive-specine bias.
Table 30: DQS improvements on non-automotive function-calling datasets via CC pipeline. AD: Ambiguity Diversity; PC: Protocol Compliance; R: Realism.
Dataset
AD
PC
f
function-call ciriooo--clllingchhaaee (orig)
0.47
0.92
0.79
function-calling-chane (aug)
0.78 (+0.31)
0.93 (+0.01)
0.82 (+0.03)
Glaive-fe-v2 (orig)
0.41
0.91
0.77
Glaive-fc-v2 (aug)
0.62 (+0.21)
0.91 (+0.00)
0.78 (+0.01)
C_alinnCllllgalalll (orig)
0.44
0.90
0.81
wwt___uuucttnnlllllll (aug)
0.70 (+0.26)
0.91 (+0.01)
0.83 (+0.02)
ClarifyVC-style augmentation consistently increases Ambiguity Diversity (AD) while keeping Protocol Compliance (PC) effectively unchanged and slightly improving Realism (R). This matches the intended behaviour: the pipeline injects structured ambiguity, rather than noise, when the underlying tools and domains unrelated to vehicles.
As shown Table 30
We then fine-tune two backbones (Qwen2.5-7B-Instte) and LLaMA3-88 (AMMA--B-IInstruct on either the original aepeline-augmentee versions of these non-automotive corpora, and evaluate all models a domain domain-neutral OOD fuzzy benchmark (OOD Fuzzy 1k). We report Function Hit Rate (FHR) in
Table 31
28

---

Published as a conference paper at ICLR 2026
Table 31: Function Hit Rate (FHR, %) on a domain-neutrer OOD fuzzy benchmark after fine-tuning on non-automotive function-calling corpora, with and without ClarifyVC-styy augmentation.
Model
Training Data
FHR (Orig.)
FHR (Aug.)
A
Qwen2.5-7B
funetion-calling-chlle
72.0
88.5
+16.5
Qwen2.5-7B
Glaive-fe-v2
71.4
85.2
+13.8
Qwen2.5-7B
Caatinn
69.1
75.4
+6.3
LLaMA3-8B
cunction-calling-Cee
68.5
80.1
+11.6
LLaMA3-8B
Glaive-fc-v2
68.0
82.9
+14.9
LLaMA3-8B
65.5
82.7
+17.2
Across all three non-automotive datasets and both model backbones, ClarifyVC-stylle augmentation yields consistent FHR gains in the range +6.3 to +17.2 percentage points, despite the fact that both training and evaluation occur entirely outside the vehicle-control domain. Together with the cross- dataset results Table 29] these experiments provide additional causal evidence that the benefits of Clarify VC originate from its SPA/AGA/FIAMMEA pipeline and the resulting data difficulty profile.
D SUPPLEMENTARY RESULTS
This section provides additional visualizations to complement the results in the main text (Section 4).
D.1 SUPPLEMENTARY VISUALIZATIONS
Comparison
175
FR, W0000
60
80
7.5
Ownn 1ng
Gened 328
Ainsion
Geibbe
Claska
Deeets (haen
Despleak
Gemst
Omna
Mucid
Figure 8: Bar chart comparing model performance across Intent Hit Rate (THR), Function Hit Rate (FHR), Parameter Completeness (F1-Score), and Protocol Compliance Rate (PCR).
Table 32 reports the zero-shot performance (%) of four large language models Qwen2.5-72B, LLaMA3-70B, Claude 3, and GPT-4 across five distinct benchmarks. Each benchmark is evaluated on four core metrics: Intent Accuracy, Function Accuracy, Parameter Coverage, and Multi-Turn Success. Overall, GPT-4 consistently achieves the highest scores all datasets, with the most pronounced advantage observed on the challenging Clarify The three open-source models show competitive performance standard tasks such as Talk2Car and CI-AVSR but exhibit sub- stantial drops on APIGen and especially Clarify VCtaata, highlighting the increased complexity and real-world variability captured by these henchmarks.
29

---

Published as a conference paper at ICLR 2026
System Prompt
sceee was use the de's billlly
canbanse shals
Acthuding MaJerow opintingg slater nutrel eautiig versnce heating crving
anead fust Surcec thot
Turn cn the new noce for wus.
foon
Frnn
you
the Vizt duslyy the HUO been tarned os.
Fleedde tw partudaly
X
Sanr,, thrds verveo tuncion todken weth the auttone parking urea so pertow rebeant operaties for you
durtere
"ammetin verry wawn sa oroning moninwnng
recwiing worvtty
Actwating vertlattis coding
Sevl polimtel---ad -tod
undestoce. terning the ventilaoon aod down fleset nos
X
Gor Tuming the wee cestistiie and wtliting thear condinaingg tored
Figure 9: Diagram showing model responses queries in various automotive scenarios.
Table 32: Aggregated Zero-Shot Performance (%) of Four LLMs on Five Benchmarks and Four Evaluation Metries
Benchmark
Metric
Qwen2.5-72B
LLaMA3-70B
Claude
GPT-4
Intent Accuracy
94.0
95.2
94.5
97.0
Function Accuracy
90.1
91.3
90.7
93,9
Talk2Car
Parameter Coverage
86.6
87.9
86.8
90.5
Multi-Turn Success
79.6
81.0
80.3
84.5
Intent Accuracy
90.5
92.3
91.0
94.2
Function Accuracy
88.5
89.0
88.8
92.0
CL-AVSR
Parameter Coverage
84.8
86.0
84.5
88.9
Multi-Turn Success
77.2
78.7
77.4
82.1
Intent Accuracy
91.5
90.8
91.7
93.5
Function Accuracy
89.0
87.5
89.4
91.5
doScenes
Parameter Coverage
85.4
84.7
85.8
88.5
Multi-Tum Success
78.4
76.8
78.8
81.5
Intent Accuracy
86.7
88.3
87.4
90.1
Function Accuracy
83.2
83.8
83.9
87.2
APIGen
Parameter Coverage
80.2
81.0
80.5
84.8
Multi-Turn Success
72.1
73.0
72.6
76.3
Intent Accuracy
72.4
74.1
73.5
77.8
Function Accuracy
67.8
68.9
68.3
72.5
ClarifyVC-Data
Parameter Coverage
63.5
64.7
64.2
69.0
Multi-Ture Success
62.9
63.8
63.4
67.0
30

---

Published as a conference paper at ICLR 2026
PG Comppesions
PSP Eovceroom
FDR Canperon)
1.0
chone
croup
Divee
chees
C0C Compartinn
Compertoon
FS5R Compuriin
cirec
Figure 10: Plot chart illustrating the comprehensive performance profile of different models across key metrics.
The bar chart (Figure highlights the superior performance the fine-tuned Qwen2.5-7B-SFFT model across basic instruction-following metrics. The response diagram (Figure(9) illustrates model behavior in automotive scenarios, while the plot chart (Figure 10] provides comprchensive performance overview.
Table 33: Advanced scemario evaluation across complex, ambiguous, and multi-turn vehicle-control instructions. Means + std over 5 runs; std < 1% for all SFT rows. Closed-source API models are evaluated once due to usage limits.
Model
IRA
PEP
FDR
CQC
DC
FESR
Qwen2.5-7B
71.5
69.4
68.9
64.7
65.4
70.3
Qwen2.5-14B
74.2
72.5
72.0
67.2
69.8
73.5
Qwen2.5-32B
78.1
75.6
75.2
70.3
73.6
77.4
Qwen2.5-72B
82.9
80.3
79.8
74.2
78.1
81.5
LLaMA3-SH
75.4
73.0
72.8
68.0
70.7
75.1
LLaMA3-70B
83.5
81.2
81.0
75.6
80.5
83.0
DeepSeek-R1
84.3
82.0
76.8
84.1
GPT4-function
88.5
86.2
85.7
80.3
85.0
87.8
GPT-4o
86.7
84.3
83.9
78.4
83.0
86.0
OpenAI-ol
86.1
83.5
83.0
77.6
82.2
85.2
Doubao
85.8
83.0
82.5
76.9
81.7
84.8
Claude
88.2
85.8
85.3
80.1
84.5
87.4
Wwen2.5-7B-SFT (Ours)
92.7 =0.5
90.5
=0.6
90.0 #0.6
85.5 +0.6
89.3
=0.5
92.0 +0.5
Qwen2.5-14B-SFT
91.8 +0.4
89.7
10.6
83.8 4.06
88.1
10.5
Qwen2.5-32B-SFT
92.1 +0.5
90.1 10.5
89.5 10.6
84.2 40.6
89.0 10.5
90.8 10.66
Qwen2.5-72B-SFT
93.0 40.5
90.2 +0.6
89.6 1-6.
83.9 4-066
90.2 +0.55
91.3 4-0.6
Table 33 reports advanced scenario evaluation across complex, ambiguous, and multi-turn vehicle
control tasks. Baseline open-source and proprictary LLMs show moderate performance: smaller backbones such as Qwen2.5-7B and LLaMA3-8B struggle with fuzzy disambiguation (FDR < 70)
and long- horizon grounding (FESR < 75). while larger backbones (e.g., Dwen2.5-728 Claude 3) achieve stronger accuracy yet incur high computational overhead. By contrast, Qwen2.5-7B-SFT. fine-tuned on Clarify VC-Data, consistently outperforms all baselines across six metrics, achieving 92.7 IRA, 90.5 PEP, and 92.0 FESR.
31

---

Published as a conference paper at ICLR 2026
Importantly, although Qwen2.5-72B attains competitive results, the gap between 7B-SFT and 72B modest (<c 6.6pp across metrics), while the computational savings are substantial: training costs drop by nearly an order of magnitude and inference latency is reduced ~10 making 7B-SFT far more practical for deployment in resource-constrained safety-critical environments. These results demonstrate that targeted exposure ambiguity-rich yet schema-compliant supervision substantially improves semantic parsing, safe clarification, and multi-turn grounding, yielding models that are both accurate and deployment-efficient compared significantly larger backbones.
LIMITATIONS
While Clarify VC-Data advances the evaluation of function call understanding in vehicle command scenarios, several limitations remain:
Modality Scope. Our benchmark primarily focuses on text-hased instruction understanding. A Although future vehicle systems often involve multimodal contexts (e-g-. vision, LiDAR, spatial audio), these are not yet fully integrated into the current benchmark version. Extend- ing CIarifyVC to multimodal grounding is an important future direction, and we are actively developing multimodal variant to enable such research.
Domain Generalizability. Although the function-call schema is designed to be extensible, current task templates are oriented toward the in-cabin control setting. Extending the dataset to cover broader domains such as driving policy, diagnostics, or V2X communication would improve general applicability.
Evaluation Reliance on Static Metrics. Our proposed metries (e.g., IRA, FDR, CQC) evaluate alignment and robustness in a static fashion. However, real-time interaction and downstream driving consequences (e.g., safety violations) are not yet modeled in evaluation pipeline.
Language Biases. As the current benchmark constructed in English, it may not generalize across linguistic or cultural variations in vehicle command phrasing. Future work can consider multilingual and dialectical command variants.
Despite these limitations, we believe VC-raay lays critical foundation for robust benchmark- ing vehicle-focused LLM deployments and opens pathways for future expansion modality, task complexity, and real-world grounding.
32


Reviews:
[{"review_id": "1kvi6tKYe4", "reviewer": "Reviewer_TGRH", "summary": "Introduces 3 novel contributions, a framework for clarifying ambiguous commands, a dataset, and an evaluation protocol. Previous works have worked with single-turn parsing, but the proposed pipeline maintains a focus on multi-turn grounding, for interaction with the user. The benchmark dataset and evaluation protocol outperform other models.", "questions": "1.  In table 2b, why was human validation conducted on ClarifyVC-Data only? \n2. Under Function Hit Rate on Page 8, what is the gold standard referring to?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Original ideas with novel approaches, high quality of the work presented and incredibly detailed, strong contributions with excellent evaluation benchmarks, the figures are very well made", "weaknesses": "Some of the comparison evaluations seem redundant and make the flow of the paper harder to read", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuxMlIZ4sf", "reviewer": "Reviewer_UDzZ", "summary": "The paper introduces ClarifyVC for clarifying ambiguous natural language commands in vehicle control, combining a dataset, fine-tuned reference models, and a multi-turn dialogue safety-aware evaluation.", "questions": "1. How does the framework ensure that “ambiguity injection” does not distort the underlying intent distribution of real user commands?\n2. How can causal relationships in model performance be evaluated and validated?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and clearly presented.\n- The benchmark is more challenging than existing open datasets, with 20k real logs with multi-turn in-vehicle commands and dialogues.\n- The improvements achieved by the model fine-tuned on the proposed dataset are strong and well-demonstrated through clear presentation.", "weaknesses": "- As the authors mentioned in the limitation section, the current benchmark focuses solely on textual command understanding. However, real in-vehicle interactions often require multimodal comprehension, where interpreting a command may depend on visual context or environmental cues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cjx90K5Hzr", "reviewer": "Reviewer_S7bf", "summary": "The paper is inspired by the ambiguity of natural-language commands for in-car voice assistants, and proposes a pipeline to augment data for the purposes of command clarification. The paper also proposes an evaluation protocol and metrics to evaluate the quality of such data and any models trained on it, including ambiguity detection and resolution in single and multi-turn dialogue.\n\nThe paper provides rigorous experimentation, examining both the efficacy of the proposed augmentation process and related work datasets (under the proposed evaluation framework and metrics), and the downstream effect of the data.", "questions": "- The paper uses the low public perception of self-driving cars as a motivating factor for this work, but it is unclear how addressing in-car voice assistants helps address this, please elaborate.\n- Please provide more details on how AD, PC and R are measured in the main body of the paper.\n- Please elaborate on the human evaluation of Section 4.1. Consulting the appendix as well, it seems that these evaluation did not consider any alternative baseline o related work. Is this assumption accurate? Without such a reference, this evaluation could be considered biased, and does not validate the method against previous work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- While not the first dataset to address command ambiguity in an in-car setting, it combines data-augmentation towards ambiguity resolution (seeded from real world scenarios) and multi-turn grounding.\n- The proposed evaluation protocol should be useful to the community moving forward, and it clearly shows the efficacy of the proposed data-augmentation process.\n- Further rigorous analysis in the paper supports the usefulness of the augmented data in fine-tuning models, including comparisons of the dataset against previous ones (under the proposed evaluation framework), evaluation of models fine-tuned with the dataset, and comparisons against closed-source models.", "weaknesses": "- The scope of the paper is limited to in-car voice assistants. While the paper claims that the approach is domain-agnostic, such is not demonstrated in this paper.\n- Unclear whether the data will be released, as they seem to be based on proprietary seeds. This could hurt both reproducibility and the impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0dCHNyQUbB", "reviewer": "Reviewer_NcpB", "summary": "This paper introduces ClarifyVC, a framework unifying data augmentation, reference models, and an evaluation protocol for in-car language control. This paper’s agent-driven pipeline creates diverse, ambiguity-rich dialogues from real-world seeded queries under schema and safety constraints. This paper evaluates single-turn parsing, conservative clarification under fuzziness, and multi-turn grounding. This paper reports gains up to 15% in parsing accuracy, 20% in ambiguity resolution, and 98% protocol compliance, supported by human evaluations of realism and applicability.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes new data, evaluation methods, and models for improving an in-car assistant—an important application.\n- It presents a set of metrics to evaluate data and model quality.\n- Extensive experiments demonstrate the effectiveness of the proposed approach.", "weaknesses": "- The task specification and the descriptions of data augmentation, training data, and test data are not detailed enough for a proper assessment.\n\n- It’s difficult to discern the exact task the paper targets from the current presentation.\n\n- The dataset is not described: the types of tasks, their variations, and their intended purposes are unclear. In particular, the real-world in-vehicle logs used in the experiments are not characterized (e.g., distribution, types, lengths).\n\n- The basic instruction-following evaluation uses both Talk2Car and real-world in-vehicle control logs, but those real-world logs are also used for training. Even if the splits differ, the distributions may be similar (the paper doesn’t specify), which could bias results in favor of the trained model.\n\n- The benchmarks used for the “Evaluation on Advanced Scenarios” are not clearly identified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)