Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks
Authors: Xingpeng Sun
Haoming Meng
Souradip Chakraborty
Amrit Bedi
Aniket Bera
Venue: Accepted by TMLR ()

Abstract: While LLMs excel in processing text in these human conversations, they struggle with the nuances of verbal instructions in scenarios like social navigation, where ambiguity and uncertainty can erode trust in robotic and other AI systems. We can address this shortcoming by moving beyond text and additionally focusing on the paralinguistic features of these audio responses. These features are the aspects of spoken communication that do not involve the literal wording (lexical content) but convey meaning and nuance through how something is said. We present ``Beyond Text''; an approach that improves LLM decision-making by integrating audio transcription along with a subsection of these features, which focus on the affect and more relevant in human-robot conversations. This approach not only achieves a 70.26% winning rate, outperforming existing LLMs by 22.16% to 48.30% (gemini-1.5-pro and gpt-3.5 respectively), but also enhances robustness against token manipulation adversarial attacks, highlighted by a 22.44% less decrease ratio than the text-only language model in winning rate.  We also present the first dataset on disfluent human audio-guided instructions for future research in this field. ``Beyond Text'' marks an advancement in social robot navigation and broader Human-Robot interactions, seamlessly integrating text-based guidance with human-audio-informed language models.

Published in Transactions on Machine Learning Research (10/2024)
Beyond Text: Utilizing Vocal Cues to Improve Decision
Making in LLMs for Robot Navigation Tasks
Xingpeng Sun
2ww.2225purd
Deportment of Computer Science
Purdue Universily
Haoming Meng
Jameng29@vvisc..oom edu
University of Maddisn
Souradip Chakraborty
schakro30@mm. edu
Department of Computer Science University of Moryland, College Park
Amrit Singh Bedi
anwritbedi@@uee..c eduu
Department of Computer Science University of Central Florida
Aniket Bera
aniketbera@purdher edu
Department of Computer Science
Purdue University
Reviewed on OpenReview: https: https://opewwsn openrevee.. net/ forum? id- ojWtg4n7Ag
Abstract
While LLMs excel in processing text in these human conversations, they struggle with the maances of verbal instructions in scenarios like social navigation, where ambiguity and un- certainty can erode trust in robotic and other AI systems. We can address this shortcoming by moving beyond text and additionally focusing on the paralinguistic features of these audio responses. These features are the aspects of spoken communication that do not in- volve the literal wording (lexical content) but convey meaning and nuance through how something is said. We present Beyond Text; an approach that improves LLM docision- making by integrating audio transcription along with subsection of these features, which focus on the affect and more relevant in human-robot conversations. This approach not only achieves 70.26% winning rate, outperforming existing LLMs by 22.16% to 48.30% (gemini- 1.5-pro and gpt-3.5 respectively), but also enhances robustness against token manipulation adversarial attacks, highlighted by 22.44% less decrease ratio than the text-only language model in winning rate. We also present the first dataset on disfluent human audio-guided in- structions /https://github com/beyond-text DENeett//IIA--A.E (eettDDDIAdataase)) for future rescarch in this field. "Beyond Tezt" marks an advancement in social robot navigation and broader human- robot interactions, seamlessly integrating text-based guidance with human-audio-informed language models.
1 Introduction
The rapid advancements in text-based Large Language Models (LLMs) like GPT-4 and the Gemini series AAciaam et al. 2023 Reid et al.) 2024) have significantly enhanced capabilities in collaborative human- robot interactions (HRI) espocially in diverse scttings, including task planning, or social navigation (Shah) lt al.] 20233. Liu
2023a Li et al.] 2023 Ren et al.) 2023 Liang al., 2023). Broadly speaking, for HRI tasks, want. to design the robot to think like human and best carry out its tasks. We

---

Published in Transactions on Machine Learning Research (10/2024)
humans interact and solve problems together, often relying verbal communication as key modality. If humans and robots interact to perform collaborative navigation tasks in large complex environments, human's 's potential unfamiliarity with the space and spatial anxiety(Chan et al. 2012 leads to providing ambiguous or uncertain navigational guidance, and these uncertaintios are not only in literal wording but also vocal nuance (Prestopnik & Roskos- Ewoldsen 2000 Golledge 2003). While LLMs exhibit. strong textual reasoning and perform adeptly in robotic tasks through textual prompt tuning, their ability to process and interpret audio information, particularly the nuanced vocal features in speech tones, remains limited (example shown in Figure 1.. Although GPT-4 Achiam et al. (2023) introduces audio-to-text API, enhancing its capability to accept audio input, this feature primarily focuses on transcription rather than analyzing vocal features. This limitation becomes particularly salient in human audio-guided social navigation, where assessing the
wrustworthiness of human guidance is crucial for successful navigation (Drbbala el al. 2021 Francis et al.
2023 Firoozi et al. )023
In this work, we present Beyond Text, an approach that incorporates both audio transcription and affective vocal features, including pitch, loudness, and duration, to improve robot navigation. Our approach lever- ages these vocal cues to interpret navigational instructions more reliably, especially when instructions are ambignous or uncertain (Sundar 2020).
Moreover, to the best of our knowledge, existing available audio-guided navigation datasets, like RxR Ku et al. 2020), do not consider vocal cues for navigation tasks. To address this gap, we introduce Disfluent Navigational Instruction Audio Datasel (DNIA). This dataset contains audio clips that capture vocal and textual uncertainties, characteristic of real-life human speech in navigational contexts. DNIA focuses on disfluencies and muanced vocal expressions, which are vital for enhancing LLMs' ability to interpret the trustworthiness of human speech. We believe that these contributions are significant first step towards developing more natural and effective LLM--ased robot-human interaction systems for social navigation.
Our main contributions are summarized as follows:
Provided Evidence for Advancing LLM Navigation with Vocal Features. The current foundation models are unable to capture the nuanced human ambiguity via only textual input. We propose the integration of affective vocal cue analysis beyond text. to let intelligent. systems like robots synthesize these streams into actionable intelligence, and thus not only reason what the human said, but also how the human said. Our results show low bias and low variance: + winning rate in detecting the uncertainty and generating appropriate next-step action and a low confidence score compared to only using LLMs to read textual transcription, indicating increased high confidence analyzing human uncertainty. Notably, all three different large language models we tested, including GPT3.5, GPT3, and Gemini Pro, exhibited a significant increase in winning rates. Beyond Text augments LLMs' ability to model human uncertainty in navigational scenarios.
A New Dataset on Disfluent Human Audio-Guided Instructions. In support of our framework and catalyze future research, we collect a dataset comprising 500 human audio
Human Guidance
LLM Decislon-Making
Streight, turn washroom, and you will sto, thestlate)" the Mail Office your (psuse)) right side (pitch ago scunds nett qu questloning)*
Current LLM
Go Straight, then rumn left at the washroom,, Maii Officee on right sidee
X
Beyond Wi
Go Striggnt, ihen can telt at the washroom, look for more information and snother person for directionn
vocca oues connor seen currest
Figure 1: Current Large Language Models (LLMs) are unable effectively interpret human vocal cues and accurately make decisions for audio audio-guided navigation involving ambiguous instructions.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Tranecription
go straight down the hall, lake the first, may be the second right to reach the destivation.
Reasoning:
Hi there, how con to the Cafeleria?
pivase. may be the socond signats language uncertainly showing the fumnn not sure which turn toko, the longer durattin of "or maybe" reinforce the uncertainty...
instruction
Text imput only (Existing)
14
Affective Analysis
-4.728
Turn feft, straight down the hall, and right
Example
and -5. 565667 Turn right, streight down the hall, and take the first right
-6.77
go straight
Duration:
Exampic Cholces
lake the second right.
Iwill follow green Instruction
"Tum left." (0.36 s)
"Go straight... 10 55 "Taks the first maybe the second righ!" (2.68
loft, straight down the hall, and continue
you get more informalion.
-1117--Ar Ask another person more procise directions to Calatoria
Pitch: change 1s Loudness: chango
Proposed
Figure 2: Beyond Text Overview: Wilh a human instruclion andio clip, we simeallaneously do branscription and vocal affective analysis. We prompt the language model to reason and generate five nert-step choices. Then, we pick the action based on the highest nert token logit probability. Note that with only the transcription model, the framework chooses the red choice. With affective analysis model, the green choice is picked.
navigational instructions with perceptual "ground truth" captured human labeling. This dataset (https://githee com tic and vocal ambiguities, offering an extensive experimental foundation for our proposed method and subsequent investigative endeavors in this space.
is characterized by a diverse range of seman-
3. Show Robustness of Vocal Cues on Improving the LLM Decision-Making: We also show the robustness of our framework against adversarial attacks by conducting token manipulation ex- periments on the prompts fed to LLMs. Under attack, our framework has 22.44% less decrease ratio than the text-only language model in winning rate. We present comprehensive ablation study evaluating the effectiveness of three specific vocal cues utilized, 15% to 20%+ increase in winning rate clearly indicating the significant role these cues play in augmenting the reasoning capabilities of LLMs. Lastly, we discuss the potential of audio augmentation in LLMs, highlighting this as a promising avenue for future research.
2 Related Works
Large Language Models for Robotics. Recent studies have demonstrated the versatility of LLMs various robotic contexts. These include employing few-shot summarizing skills for task planning in robots
(Wu et al. 2023 Liu ct al. 2023b), interpreting contextual information for robotic locomotion dccision-
making (Shek et al. 2023 integrating visual feedback for open-world robotic manipulation (Lin et al.
2023a: Ahn et al. 2022) and enhancing motion planning in autonomous driving Mao et al. (2023). For human-robot interaction, Shah et al. (2023) suggests using LLMs for semantic prediction of object locations, thereby optimizing navigation. [Ah et al. (2022) introduces multimodal input to generate the optimal long-horizon instructions and effectively score the affordance probability of the instruction for robotic task planning. Mcanwhile, Ren et al. (2023) develops a framework assess and synchronize the uncertainty levels in LLM-based planning systems, enabling them to seek assistance when necessary. However, unlike our framework, Ren et al. (2023) does not account for potentially unreliable human input but rather relies on human selection among generated options. Beyond Text uniquely concentrates on assessing human uncertainty through affective computing for robotic navigation, particularly when guided by human audio that might be ambignous or incorrect. This approach allows robots to gauge the reliability of human-audio navigation instructions by analyzing vocal emotions, enabling more informed reasoning.
Foundation Models with Audio. The ability to perceive and interpret auditory information is essential for AT agents in the real world. GPT-4's recent. (phate(Achiam Achiam al. 2023), incorporating Whisper's audio-
3

---

Published in Transactions on Machine Learning Research (10/2024)
to-text feature [Radford et al.
significantly advances foundational models' capacity to transcribe audio inputs. However, this transcription process often overlooks nuanced vocal features inherent in speech. Ghosal et al. (2023); Borsos et al. (2023) explore language models for generating audio from text and enhancing visual understanding in vidcos. Notably, Tang et al. (2023) integrates speech audio encoders with a multimodal language model, enabling direct processing of audio inputs for lasks like speech translation, answering spoken queries, etc. Despite these advancements, current research does not specifically address the use of audio inputs in aiding human-robot navigation or in extracting vocal uncertainty from speech.
2023
Uncertainty Modeling in LLMs. Gauging the trustworthiness of responses generated by LLMs remains an open challenge in today's LLM cra, with limited rescarch on uncertainty quantification for natural lan- guage generation. Lin et al. (2023b) compared multiple uncertainly measurements in the black-box language model, and Kuhn et al. (2023) introduced entropy as a method to model uncertainty in the large language model. In our approach; we take inspiration from prior works (Kuhn et al.) 2023 Lin et al.) (023b) and define a confidence score C(p) that is inversely proportional to the Kullback-Leiln divergence between the current LLM's prediction and ground truth human perception distribution show the LLM's confidence on uncertainty measurement.
Human Affective Analysis in Robotics. For social robots to effectively coexist and interact with humans, it is imperative that they comprehend human emotional states for decision-making processes. Emotion understanding from speech for human-robot interaction has been studied in Lakomkin et al. (2018a) while deep reinforcement learning methods (Dorbala et al., 2021) and cognition model (Eppe et al. 2016) has been used understand textual ambiguities in natural languages. RxR dataset (Ku et al.) 2020 audio dataset for visual language navigation but does not cover the affective cues embedded in the vocal instructions, which hold substantial information that can reveal human uncertainty and trust. Beyond Text introduces an andio-angmentation technique for large language models (LLMs) that analyzes textual and vocal cues in human audio commands. This approach enables more human-like like robotic systems for social navigation by identifying emotions and ambiguitics through both speech content and vocal tones.
well-known
3 Our Approach
In this section, we first mathematically characterize our novel framework, 'Beyond Text,' which uses LLMs to model human trust and uncertainty based on audio-based navigational guidance. As shown in Figure we consider a setting where robot asks the human for instructions, and then the human provides instructions in the form of audio instruction. This setting is adapted based on prior work on social navigation (Hu et al. 2019 Lakomkin et al. 2018b Eppe et al.] 2016 Francis et al. 2023). In order to answer the question raised by the robot, the human replies with a voice prompt v E V, which belongs to set of audio responses V, which is then converted to a textual description using the best available transcription model Tm V - (here S is the set of all possible sentences). Simultaneously, is mapped to an affective cue set K. via an affective cue model AC: : V + K. The combination of textual and affective cues is denoted as:
@(V) = K
(1)
which constitutes the prompt for the LLM (denoted by po with 0 being the language model parameters). which outputs a response y = [y1. , .yk] by generating each token y at a time i. The joint probability over the tokens can be written as:
p(())(())= = II po (yi v1, y2. 11
(2)
To perform decision-making for robots, with a few-shot prompt that includes possible next steps in a few scenarios, the LLM generates candidate responses { and chooses the most likely answer. We have briefly deseribed it in Figure 2 and will provide more details below.
3.1 Semantic Uncertainty Quantification
The process begins by converting the audio clips of navigational guidance into text using the open-source Whisper model (Radford et al.] 2023), which we explain in detail in Appendix B The transcribed text has
4

---

Published in Transactions on Machine Learning Research (10/2024)
two key functions for the LLM: identifying uncertainty-rellteed semantic cues and formulating appropriate navigational steps. We focus on disfluency in human speech as an indicator of uncertainty, which affects the confidence measure of the LLMs (ef. 3/. We categorize language uncertainty into three types as follows:
(1) Textual Uncertainty: Phrases like "probably," "may," " "might," and "I think" within the transcribed text suggest a lack of conviction (Dorbala et al. 2021
(ii) Speech Repair: In instances where individuals correct themselves (Lou et al., 2019), such as saying, "Take right-uhh, mean take left and go straight," it is crucial to parse and interpret these corrections to avoid misdirection.
(iii Hesitation Signals: Pauses or hesitations, as in "Go straight and, err, take the second left," " may siguify uncertainty regarding subsequent directions (Ogata et al. 2009), despite apparent confidence in precoding instructions.
These textual patterns are typical in human communication when conveying uncertainty. We prompt the LLMs to recognize these cues, subsequently adjusting the confidence level in the reliability of the instructions that follow these indicators.
3.2 Vocal Affective Cue Analysis
In addition to textual content, the prosody of spoken instructions can be indicative of huran uncertainly
(Mittal et al. 2020 Goupil et al. 2021 Jiang & Pell 2017 Guyer et al.) 2019). Our analysis focuses on three primary auditory features captured within the Mel-frequency copstral coefficients of the voico waveform
detect potential uncertainly:
Duration: When individual is unsure about the instruction, the speed of their speech either slows down as they are trying to think through what to say or speeds up as becoming more nervous (Jiang & Pell 2017 Székely et al. 2017). The speech rate is a vocal characteristic indicating certainty tone change (Guyer ett al. 2019). We measure the duration of each instruction segment inside the audio and evaluate whether the duration has drastically changed, changing it to too long or too short for specific segments. For instance, for recording that each instruction phrase as short as second, an elongated phrase like "or the second right" with duration of 5.68 seconds can indicate hesitation and diminished certainty.
Pitch: A common feature of uncertainty is rising intonation at the end of phrases sentences, which can sound as if the individual is asking a question rather than making a statement (Goupil et al.) 2021 Guyer et al.) 2019). We identify the most unusual pitch change pattern in the waveform and align it with the instruction piece by timestamp.
Alongside pitch, the uncertainty can also be embedded in loudness, since unpredictably rising and falling volume can indicate hesitation or self-doubt (Goupil et al., [021]] Székely al.) (017).. Similar to pitch, only detect the most drastic loudness change in the audio clip and align it with the instruction piece by timestamp to infer possible uncertainty of instruction pieces.
Loudness:
To quantitatively measure pitch and loudness features, we design a lightweight yet effective algorithm (see Algorithm [ in AppendixF) to detect the max change of these two features in the audio clip. For the duration, we split the transcription sentence into sub-instructions and measured the duration of each sub- instructions. We only mark the sub-instructions that are significantly longer than other sub-instructions the same audio clips as uncertainty signals. Force alignment, process that synchronizes text. fragments to their corresponding segments in audio narration, is used to align the vocal features' timestamps with the sub-instructions. Examples are shown in Figure 2 and detailed in Appendix C
In this work, we focus on siguificant changes in these audio features that exceeded a predetermined threshold while excluding variations in the initial and final three seconds recordings to avoid contamination by recording noisc. This audio augmentation prompt serves as input for decision- decision-making processes that leverage language models. In a way, we give the LLMs the car to listen to human affective cues embedded inside the vocal clip, thus improving their reasoning skills.
5

---

Published in Transactions on Machine Learning Research (10/2024)
All
Vocal Uncerta inty
Language Uncertainty
Uncertainty
Language Jncertainty
2.30
1.6
1.75
1.359
1.434
1.4
1.135 1.119 1.145
3.25
1.50
0.999
1.07
0.931
Contidence Score
1.0
1.2
C.778
0.765
0.788
0.955 0.955 0.955
0.806
0.726
0.65
0.8-
0.75
0.6
0.50
0.4
0.25
0.2
0.30
Transcript Only
With Reasoring
With Reassnn and Vocal Cue (Ours)
0.0
Transcript Only
With Reasoning
With Reason and Vocal Cue (Ours)
Context
Context
a)
b)
Figure 3: a) Averuge Confidence Score by contezt and audio type for samples where the language model picks the same choices as human perception. The skinny bar the error bar, representing the standard deviation of the confidence score. Error bars give general idea of how precise a measurement is or, conversely, how for from the reported value the true (ervor-free) value might be. b) Average Confidence Score by context and audio type. An overall low variance is indicated by the increased average confidence.
3.3 In- mn-Context Prompting
Beyond Text enables the language model to exploit its inherent reasoning capacities for strategic planning and decision-making tasks through In-Context Learning (ICL) (Yousefi et al., 2023 Liu et al. 2023a
ct al. 2023), by providing only few-shot examples with no explicit training. We illustrate ICL (Yousefi et al. 2023 and Chain-of- oboin-of-Thought (Wei et al. 2022) by including three comprehensive examples that delineate all potential scenarios a robot might encounter while navigating human interactions for directional assistance. Specifically, we provide examples that: only has language uncertainty; 2. only has vocal uncertainty; 3. has both semantic and vocal uncertainty signals. In the response phase, the model reason to suggest spectrum of five potential actions. Notably, the first option (A) is always a paraphrased version of the transcription sentence without uncertainty, mimicking the scenario where the robot chooses to blindly follow the human audio-guided rustroci and the final option (E) is consistently presented as "ask another person nearby for direction," ensuring a fail-safe in the decision bocision-making process. Detailed prompts and LLM output examples are shown in the Appendix C
3.4 Scoring Choices by Polling LLMs
Each step in an LLM generated plan denoted as y, comprises sequence of symbols (01,02,-.,,0, Traditional LLMs calculate the joint probability of these sequences p(y) = IT p(0, (0,-.., (ah....,0-- 0;-1). However, this approach is flawed due to its high sensitivity to the sequence length n, rendering p(y) unreliable scoring function. To mitigate the length bias, we introduce multiple-choice Q&A framework for planning. This method refines the planning process to select the most probable next token from predefined set of labels Y = {A, B, C, D, E}. This results in a probability set p(Y) = p(A),p(B)) p(C'),(( ((D,((()) ((D),(()) from which we select the label with the highest probability, maxx((Y) as the optimal action. By comparing the confidence change AC C(peocat) C(prest), we assess the impact of vocal prompts on the pocision-makinn process of LLMs.
4
New Dataset: Disfluent Navigational Instruction Audio Dataset (DNIA)
DNIA is a dataset comprising a diverse collection of human audio recordings. All the audio clips are grounded in the robot navigation settings, as we ask the volunteers to give instructions to a location inside the buildings they have been to. These recordings capture spectrum of navigational disfluencics, including 500 audio clips contributed by twenty individuals, including ten males and ten females, and encompassing wide array of native and non-native speaker vocal styles. Each clip, ranging from 10 to 30 seconds, includes series

---

Published in Transactions on Machine Learning Research (10/2024)
All Audio Sample
Vocal Uncertainty (VU)
Language Uncertaintyy (LU)
160
140
140
100
100
Count
Count
60
60
20
20
A
E
A
8
A
Varseription Only
With Reason
With Reason and Affective Cue (Qurs)
Figure 4: Distribution of top log-probability choices by contert and audio type. Our work (green) shows a dy- namic change from Transcription Only (Blue). Instead of blindly choosing "A" and following the instruction with uncertaindy, Beyond Text chooses "B", "C", "D", or "E", which successfully identifies the uncertainty component within the audio.
of English directives necessary for navigating to specified location. The audio files in WAV format, normalized with headroom of -2dB, facilitating ease of use and consistent audio quality.
We identified and categorized two types of disfluency within our dataset: language uncertainty (LU): 285 data clips and lext correct by vocal tone uncertainty (VU): 215 data clips. The LU calegory is the audio that contains semantic disfluency signals, such as hesitation and language uncertainty. The VU category is particularly noteworthy, as encompasses instances where the textual transcription alone does not manifest any indicators of uncertainty, but the vocal tone shows sigmificant uncertainty clues. Regarding ethics and privacy concerns, our dataset is collected by a group of fellow students with written consent to anonymously publish the recordings of their voices for research purposes. We reviewed all the audio files to ensure personal information is present, and the annotations (including audio filenames) can not be matched to a specific participant. Authors have human subjects training certificates from the university. Therefore, we follow the existing guidelines to ensure the DNIA dataset does not have any ethics issues, and we will publish the dataset soon after the review process for research purposes. Table shows examples for the audio prompt in DNIA, more details can be found in Appendix
To mimic real-world Human-Robot Interaction (HRI) scenarios closely possible, align with the methodologies suggested in LLM evaluation literature (Bang et al.) 2023 Chang et al. 2023). We employ human annotation as one of the primary assessment techniques. This involves manual evaluation, which is particularly relevant in HRI contexts due to its ability to capture the nuanced complexities of human interaction. We use human evaluation to calculate the winning rate analyzing human navigational guidance uncertainty in Section 5.4] Our approach entails labeling hurnan audio-guided instructions through selection process from a of five multiple-choice options designed represent the uncertainty inherent in the content accurately. We recruit different set of 20 participants with diverse genders and dialect backgrounds to ensure broad range of perspectives. We developed participants listened to navigational audio clips. The instructions emphasized careful listening and unbiased selection, with constraint of a maximum of one minute per clip to simulate real-world Docissionmmkiing time pressures.
study website (see Appendix H where these
Table 1: Example Audio Clip Content in DNIA Dataset
Type LU
Example Content.
Explanation
"Go straight, maybe turn left cafe shop"
'Maybe' uncertain word
VU
"Go straight, turn (hesitate) left cafe shop (pitch change)"
Textual correct but voice sounds uncertain

---

Published in Transactions on Machine Learning Research (10/2024)
5 Experimental Results
To validate the efficacy of our approach, we utilize confidence score and winning rate to show that Beyond Text can achieve low variance and low bias results compared to current LLM-based human instructions without vocal cues. We will first introduce the evaluation metric, and the comparison experiments, ablation studies on various vocal cues and LLM types.
5.1 Evaluation Metric: Confidence measure
In this paper, we specifically focus on improving the uncertainty or the confidence score of the LM py on the optimal (ground-truth) candidate responses y"".
We note that as defined in equation 1 our model is given by po(-)Q(u) where Q(v) is combination of vocal and text cue, For a given Q(v), let us consider {y' { as the set of possible candidates, and the probability of each candidate is given by Q(()))(())) and define the probability distribution p as p == .P probability distribution across possible candidates.
po(v"(Q(m)) po (rr)) ... Po (v' [Q())). Hence, this is clearly interpratable in the sense that it is the
We do not assume that the gound truth distribution pa known or have any specific form but holds that p" ("(y") > for all j so that it valid distribution and E, ("(())) = 1. In most of the practical cases, the ground truth distribution will be concentrated around the true response yd if it is unique. Now, we define the confidence measure C(p) of a distribution as
C(p) KI(P,p")
(3)
where we note that the confidence measure will reduce as the KL divergence between p and pa increases. The the range of a confidence measure, which is inversely proportional to entropy, bounded between [0, oo). Ideally, we would like have the confidence measure as high as possible, which requires us reduce K pT) We first expand and note that
=
Ee(rr)
> ((rr) log(p" (v')),
(4)
where H=p)= E,p:==E,<>>> log(p(y")))
p(V")
Notably, higher confidence score which indicates lower KL divergence between p and p* desirable, as it signifies a closer alignment with the ground truth. In this work, we empirically show that adding the vocal cue in the input helps to reduce the KL(p,p") by plotting the confidence measure equation for robotic navigation tasks in Section
5.2 Confidence Score Improvement
As discussed in Section 5.1 we show that our method achieves a low variance with an increased confidence score. To better visualize the low variance of our approach, we first selected subset of 100 data instances wherein the choices made by Large Language Models (LLMs) were in alignment with user selections, shown in Figure Then, analyze the average confidence score across all 500 data clips in the dataset. to demonstrate the efficacy and generalizability of our framework, shown in Figure It reveals marked increase in confidence score: Beyond Text significantly surpasses the text-only language model, both the transeription-only and the LLM using Chain-of-T Thouught (CoT). This trend was also observed across two distinct categories. The introduction of affective cues significantly lowers variance, enhancing existing LLMs ability to interpret human vocal uncertainty with confidence.
5.3 Choice Distribution Change
Figure shows the count of top log-prob choice for each audio clip in our datasct. As audio clips contain various types of disfluency, we want the framework eventually pick from choices (D), that iden- tify the uncertainty within the audio and handle these uncertainties accordingly (choice structure discussed
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Ablation Study for Winning Rate based on Various Vocal Cue
Vocal Cue
Winning Rate
Without Reasoning
With Reasoning
Pitch
Loudness
Duration
All
VU
LU
All
VU
LU
12%
22.79%
21.75%
49.30%
36.74%
58.60%
x
X
38. 58%52%
40.00%
37.19%
61.68%
64.65%
59.30%
V
42.31% %
43.72%
41.05%
63.07%
61.40%
64.21%
*6
*
V
34.33%
36.74%
33.28%
60.88%
61.40%
60.35%
V
x
40.32%
39.53%
40.70%
67.47%
70.23%
65.26%
V
37.52%
38.60%
36.49%
64.87%
66.51%
63.51%
*
40.72%
42.79%
38 38.95%
65.47%
67.91%
63.51%
4
36.93%
40.00%
34.39%
70.46%
72.56%
68.77%
Table 3: Winning Rate by Context and Audio Type
Winning Rate
Method
All
VU
LU
Transcription Only LLM
22.16%
22.79%
21.75%
With Chain-of-Thouuht Reasoning
49.30%
74%
58.60%
Ours
70.46%
72.56%
68.77%
in Section 3.3), such as looking for more information or asking another person the location point where human voice sounds unsure.
The blue bars mostly accumulated choice (A) (220 out of 500), meaning that with only transcription, the LLM tends to blindly follow the navigational instruction without identifying uncertainty in the instruction, which is undesirable. With reasoning in in-context learning, the distribution shifts, and with the affective cue audio augmentation, the data that result top log-prob on choice (A) becomes even lower (15 out of 500). The shift in choice distribution qualitatively shows the effectiveness of our method choosing the next-step action that handles human vocal uncertainty.
5.4 Winning Rate
Besides indicating low variances and multiple choice distribution change, we also calculate the winning rate. We compare the human annotations for each DNIA data clip with the highest bog-probabilityy choices generated by the model to ascertain whether the choices and reasoning processes generated by the LLMs are in harmony with human conceptual understanding.
Nauce
Winning rate
(5)
Niotal
where Nsuce represents the number of instances where the LLM selects option that aligns with the DNIA dataset's human annotations, and Neotal denotes the total number of audio clips that prompt the LLM. Essentially, the winning rate is analogous to a success rate, ranging from 0% to 100%, with higher winning rate indicating better performance.
The winning rate reported in our paper is calculated over the entire DNLA dataset, which consists of 500 audio clips: 285 for LU and 215 for VU.
The results, detailed in Table 3 show the limitation of current LLMs in explaining vocal uncertainty from only textual input and the effectiveness of our audio augmentation approach. Given that the audio clips originated from human speakers, the most reliable measure of their trustworthiness was human perception, hence the incorporation of a user study to gange the winning rate with which the language model identified uncertainties in the audio clips and selected the appropriate next-step action.
Table 3 demonstrates that our method exhibits reduced bias, outperforming both the singlo-modal transcrip- approach. Integrating vocal affective cues, which allow LLMs to
tion method and the
9

---

Published in Transactions on Machine Learning Research (10/2024)
Transcripiion Model
Tum left straight hall, first or may be the
Exemmll Exampie Choices
Paraphrase the text to soundss very confident
second right to reach the destination
Affective Analycis Model
-4.728 Must defentrlly go straight the right certainly.
instruction
Only taxt imput (Existing)
Duration:
"Tmm loft" anconds)
-0.399 urn left, go straigt down the hall, and continue until you qet more information.
"Go streight... " (0 55 secor da)
"Take the first or maybe second right (2.68 seconds)
11.197 Ask another person for more precise
directions Cafeteria.
will follow the green instruction
Phtck: change 2.15
Loudness: change 2 9s
Our Approach
Figure 5: The Adversarial Language Model Atlack Pipeline. An adversarial otlack that paraphruses the input text to sound very certain by deleting tertual uncertainty signals is applied.
process how statements are spoken, markedly enhanced performance. The overall winning rate surged to 70.46%, with a pronounced improvement in VU interpretation, evident in a 72.56% winning rate.
Table 4: Winning Rate Different LLMs
Winning Rate
LLMs
All
VU
LU
GPT-3.--turbo--nstruet (Transcription Only)
22. 22.16%
22.79%
21.75%
(Beyond Text)
70.46%
72.56%
68.77%
text-davinci-003 (Trranscription Only)
57 57.88%
57.67%
57.89%
text-davinci-003 (Beyond Text)
69.06%
69.77%
68.42%
Gemini--1.-Pro (Transcription Only)
43.11%
41.40%
44.56%
Cemini-1.5-Pro (Beyond Text)
65.27%
66.98%
63.86%
5.5 Ablation Study
Vocal Cue: To elucidate the impact of individual components within the vocal cue framework, we conduct comprehensive qualitative winning rate study by isolating each category of vocal cues and presenting them exclusively to the language model. We report the vocal cue ablative study in Table 2 In the absence of reasoning, the highest winning rate was observed with the exclusive use of loudness features. This suggests that incorporating multiple vocal features without reasoning may overwhelm the model, hindering ability to correlate vocal characteristics with uncertainty. Conversely, with reasoning integrated, vocal cues significantly strengthened the LLM's proficiency in interpreting vocal uncertainty and facilitating robotic navigation, reflected in the increased winning rates.
LLM Type:
To show the generalized ability of Beyond Text, we test it on three different popular LLMs. including GPT3.5, GPT 3, and Gemini. Based on the result shown in Table g we conclude that Beyond Texl improves the winning rate on all three LLMs significantly. Details about parameters for each type of LLM is shown in Appendix
6 Robustness to Adversarial LLM Attacks
With the advancement of language models, safety concerns such as adversarial attacks and token manip- ulation prompts have become increasingly prominent. Adversarial attacks or token manipulation prompts Ribeiro et al. 2018 Shayegani al. 2023) could potentially trigger the model to output something unde-
sired Zou al. 2023 Greshake al. 2023 Given piece of text input containing sequence of tokens.
10

---

Published in Transactions on Machine Learning Research (10/2024)
can apply simple token operations like replacement with synonyms to trigger the model to make the incorrect predictions (Morris et al. 2020 Li et al. 2020).
Table 5: Winning Rate by Context and Audio Type (Under Token Manipulation Attack)
Winning Rate
Method
All
VU
LU
Transcription Only
22. 22.16%
22.79%
21.75%
Transcription Only (Attack)
9.78%
11.23%
7.91%
Decrease (in Percentage)
55.87%
50.72%
63.63%
Ours
70.46%
72.56%
68.77%
Ours (Attack)
46.90%
50.23%
44.21%
Decrease (in Percentage)
33.43%
30.77%
35.71%
We demonstrate that Beyond Text exhibits robustness against token manipulation adversarial attacks tar- geting LLMs. Our attack pipeline is designed as follows: Post-transcription, a paraphraser alters the original transcription (71) into a new version (T2), systematically replacing uncertainty semantics with deterministic language. We then adjust the prompt to align the vocal and textual information with 72. Subsequently, the LLM is instructed to replicate 72 as an option (A) instead of generating a freeform paraphrase (71') devoid of uncertainty cues. This approach highlights the limitations of current LLMs, which tend over-rely on semantic textual instructions with no clue about the nuances of vocal delivery. Figure illustrates the attack pipeline A detail example is shown in Appendix D
Table illustrates the comparative robustness of Beyond Tezt. While the transcription- oonllyttoooonly approach ex- hibits a significant performance decrease of 55.87% under attack, Beyond Test, with its audio augmentation preserving vocal cues, exhibited a notably lower reduction of 33.43%. This suggests that audio augmenta- tion in Beyond Text enables L.Ms to maintain reasoning capabilities and resist. being misled by text-based adversarial attacks.
7 Conclusions and Open Questions
In our work, we highlight the shortcomings of current LLMs in robotic navigation, specifically their inability to compute uncertainties in human audio instructions. This issue largely stems from the reliance andio transcriptions in state-of-the-art methods, leading to loss of critical audio features such duration, pitch, and loudness. Therefore, we state our position that we cannot just reply to the transcriptions of the human audio descriptions but should instead also focus on the audio features. To this end, we introduced Beyond Tezt, a novel approach that integrates the affective cues from the human voice and provides empirical evidence that significantly enhances the LLMs' ability to interpret human vocal uncertainty. We developed the DNIA audio dataset to push forward the research in this direction. Notably, Beyond Text outperforms existing text-based language model techniques in terms of win rate and entropy confidence, demonstrated in user study. This represents a progression in the development of sophisticated audio-augmented LLMs. It underscores the importance of incorporating affective audio computing capabilities in L.MMs, particularly for enhancing social robotics navigation tasks.
Questions Regarding Setup. While explicit disambiguation, such as asking an additional question, could indeed clarify instructions, it is not always feasible real-time interactions where immediacy is crucial. Our approach designed to allow robots to infer potential uncertainties based on vocal cues, enabling them act more autonomously and adaptively in dynamic environments. This is particularly important in real-world scenarios, such as theme parks or shopping malls, where visitors are often unfamiliar with the environment. As first-time visitors, they might provide uncertain or incomplete answers due to their lack of knowledge. In such settings, robot cannot rely on repeatedly asking for clarification to find a confident response. Instead, our method allows the robot to navigate uncertainty and make informed decisions without the need for continuous human interaction. While it is true that some existing studics focus on having robots directly re- ask humans align uncertainty, such as in Ren al. (2023), our work serves as a complementary approach. We specifically target the robot's ability to infer uncertainties based vocal cues alone, without the need for
11

---

Published in Transactions on Machine Learning Research (10/2024)
explicit clarification. This focus on vocal cue-based uncertainty detection is crucial in enhancing the robot's ability to detect genuine uncertainly, leading to more intuitive and seamless human-robot interactions. We acknowledge that disfluencies can arise from external factors, such as distractions, but our method is designed to help robots discern when such disffuencies are indicative of genuine uncertainty, thereby improving their decision-making process without relying on continuous human input.
Open Questions. Next, we emphasize several critical questions and directions pertaining the research audio augmentation to improve LLMs' ability to understand human speech, which are still unsolved and need further discussion.
(1) Improving Quality of Audio Augmentation: It is essential to consider more audio features, such as intonation and prosody, with learning-basee baaid affective computing methods to improve the quality of audio augmentation. The DNIA is a foundational dataset designed for studying human uncertainty guided navigation tasks, but can also be used in other HRI-related tasks. In the future, the dataset will be expanded through the addition of more related vocal features. Additionally, the relation between different vocal features also needs to be carefully articulated for different tasks that LLMs can provide concrete reasoning. As we stated the position that affective cue analysis is currently missing from existing LLM, and beyond text is an audio augmentation method that can help LLM hear not only 'what' people said but also 'how' people convey their ideas; it is also excited to explore the potential of vocal features in improving mudio-to-text tasks, such as video emotion understanding, etc.
audio-
(2) Grne-Grained Definition of Uncertainty: Beyond Text marks a notable first step in interpreting human uncertainty in human-robot. conversations, but it has limitations. For instance, a directive like "Turn left at the washroom. Oh, sorry, turn right at the washroom' could signify either uncortainty or corrected command, highlighting the complexity of human speech. It is crucial to address such uncertainly in a fundamental manner in future research.
(3) Advancing LLMs to Listen Audio without Text: While we currently rely on textual transcriptions, emerging research, notably Tang el al. (2023), demonstrates LLMs processing audio directly with considerable success in speech tasks. However, these models still lack the capability to fully grasp human emotional nuances, especially in conveying uncortainty.
(4) Integration with Real World Social Robotics: Beyond Texl introduces human audio-guided framework tailored for social robotic navigation, designed to discern human trust from a robotic perspective. However, it is important to acknowledge that the real-world deployment of such a model faces additional complexities, particularly due to the presence of multiple audio sources beyond human speech, which request the attention of the research community.
Broader Impact Statement
This research has the potential to advance our understanding of vocal cue interpretation, particularly in fields such large language models, audio uncertainty quantification, and human-robot interaction. However, it is crucial to acknowledge and address potential biases that may arisc in interpreting vocal cues. These biases could stem from factors such as gender, accent, or cultural background, which may influence how vocal signals are perceived and understood. While our approach is effective, our approach is restricted by the capability of current state-of-tho-art Large-Language Models. Therefore, it should be applied thoughtfully and is more appropriate for less sensitive situtio to reduce the risk of biases impacting downstream tasks.
Acknowledgments
We are doeply grateful to Bufan Xiao, Xuanting "Fina" Zhou, Yutong "Chloe" Wu, Tongjia Zhang, Muqing Tang, Linqing Liu, Xindi Tang, Phu Pham, and Hrishikesh Viswanath for their invaluable contributions the DNTA dataset. collection and their assistance in revising this paper.
12

---

Published in Transactions on Machine Learning Research (10/2024)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl arXiv:2303.08777 08774, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not i say: Grounding language in robotic affordances. arXiv preprint or arXiv:2204.000 016611, 2022.
Yejin Bang, Samuel Cahyawijaya, Naycon Loe, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, al. multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint Xrviv:330 arXiv:2302.040R3, 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Do- minik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Edgar Chan, Oliver Baumann, Mark A Bellgrove, and Jason B Mattingley. From objects to landmarks: the function of visual location information in spatial navigation. Frontiers in psychology, 3:304, 2012.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2023.
Vishmu Sashank Dorbala, Arjun Srinivasan, and Aniket Bera. Can robot trust you?: A drl-based ap- proach to trust-driven human-guided navigation. In 2021 IEEE International Conference on Robolics and Automation (ICRA), pp. 3538 3545. IEEE, 2021.
Manfred Eppe, Sean Trott, and Jerome Feldman. Exploiting deep semantics and compositionality of natural language for numan--root-interacttio In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 731-738. IEEE, 2016.
Roya. Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, al. Foundation models in robotics: Applications, chal- lenges, and the future. arXiv preprint arXic:2312.0788 07844, 2023.
Anthony Francis, Claudia Pérez-d' Arpino, Chengshu Li, Fei Xia, Alexandre Alahi, Rachid Alami, Aniket Bera, Abhijat Biswas, Joydeop Biswas, Rohan Chandra, et al. Principles and guidelines for evaluating social robot navigation algorithms. orXiv preprinl arXio:2306. 16740, 2023.
Deepanway Ghosal, Navonil Majumıder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction-tumed Ilm and latent diffusion model. arXiv preprint arXit:2304.13991 2023.
Reginald G Golledge. Human wayfinding and cognitive maps. Colonization of unfamiliar landscapes: the archoeology of adaptation, 25, 2003.
Louise Goupil, Emnanuel Ponsot, Daniel Richardson, Gabriel Reyes, and Jean-Julien Aucouturier. Listeners' perceptions of the certainty and honesty of a speaker are associated with a common prosodic signature. nature communications, 2021.
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. More than you've ve asked for: A comprehensive analysis of novel prompt injection threats to inplicaationnnnegraate large language models. Xix e-prints, pp. arXiv-2302, 2023.
Joshua Guyer, Leandre R Fabrigar, and Thomas Vaughan-Johnstonn Speech rate, intonation, and pitch: Investigating the bias and cue effects of vocal confidence on persuasion. Personality and Social Psychology Bulletin, 45(3):389 103399959----1 2019.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Zhe Hu, Jia Pan, Tingxiang Fan, Ruigang Yang, and Dinesh Manocha. Safe navigation with human instruc- tions in complex scenes. IEEE Robolics and Aulomalion Letters, 4(2):753 2019.
Xiaoming Jiang and Mare D Pell. The sound of confidence and doubt. Speech Commumication, 88:106-126, 2017.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprinl arXiv:2010.0000 2020.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncortainty: Linguistic invariances for uncer- lainty estimation in natural language generation. International Conference on Learning Representations (ICLR), 2023.
Egor Lakomkin, Mohammad Ali Zamani, Cornclius Weber, Sven Magg, and Stefan Wermter. On the ro- bustness of speech emotion recognition for human-robot interaction with deep neural networks. In 2018 TEEE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp. 854-860. IEFE, 2018a.
Egor Lakomkin, Mohammad Ali Zamani, Cornelius Weber, Sven Magg, and Stefan Wermter. On the ro- bustness of speech emotion recognition for human-robot interaction with deep neural networks. In 2018 TEEE/RSJ International Conference Intelligent. Robols and Systems (IROS), pp. hTt.://.RR........9....9
2018b. doi:
Boyi Li, Philipp Wu, Pieter Abbeel, and Jitendra Malik. Interactive task planning with language models, 2023.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-altack: Adversarial atlack against bert using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193 6202, 2020.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493 9500. IEEE, 2023.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mo- hammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.0755 2023a.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187, 2023b.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p Empowering large language models with optimal planning proficiency. arXiv preprint (xXt::2304.11/77, 2023a.
Zeyi Liu, Arpil Bahety, and Shuran Song. Reflect: Summarizing robot. experiences for failure explanation and correction. arXiv preprint arXi::2306.15 15724, 2023b.
Paria Jamshid Lou, Yufci Wang, and Mark Johnson. Neural constituency parsing of speoch transcripts. arXiv preprint (0tt::990400555 2019.
Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.0144 15, 2023.
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emotions don't 't lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM international conference on mullimedia, pp. 2823 2832, 2020.
14

---

Published in Transactions on Machine Learning Research (10/2024)
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing: System Demonstrations, pp. 119 126, 2020.
Jun Ogata, Masataka Goto, and Katunobu Itou. The use of acoustically detected filled and silent pauses in spontaneous speech recognition. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4305-4308. IEEE, 2009.
Jillian Prestopnik and Beverly
The relations among wayfinding strategy usc, sense of direction, sex, familiarity, and wayfinding ability. Journal of environmental psychology, 2772):177-111, 2000.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Nlya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 18492-285111 PMLR, 2023.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwicser, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens context. orXiv preprint arXiv:2403.055: 2024.
Allen Ren, Amushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, al. Robots that ask for help: Uncertainty alignment for large language model planners. In Proceeding of 2023 Conference on Robol Learning (CoRL 2023), 2023.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Annual Meeting of the Association for Computational Linguisties, 2018. URL
https://app.sematiiicciit
Dhruv Shah, Michael Robert Equi, Blažej Osiáski, Fei Xia, Brian Ichter, and Sergey Levine. Navigation with large language models: Semantic guesswork as a heuristic for planning. In Conference on Robot Learning, pp. 2683-2699. PMLR, 2023.
Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh, Plug and pray: Exploiting off-the-shelf components multi-modal models. arXiv preprint arXio:2307.44533 14539, 2023.
Chak Lam Shek, Xiyang Wu, Dinesh Manocha, Pratap Tokekar, and Amrit Singh Bedi. Lancar: Lever- aging language for context-aware robot locomotion unstructured environments. arXin preprint arXiv:2310.00/81 2023.
S Shyam Sundar. Rise of machine agency: A framework for studying the psychology of human ai interaction (hail). Journal of Computer-Mediotted Communication, 25(1):74-88, 2020.
Eva Székely, Joseph Mendelson, and Joakim Gustafson. Synthesising uncertainty: The interplay of vocal effort and hesitation disfluencies. In INTERSPEECH, pp. 804 808, 2017.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang, Salmonn: Towards generic hearing abilities for large language models. arXiv preprint
arXiv:2310. 13289, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Demny Zhou, et al. Chain-of thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-2444 -44837, 2022.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng. Shuran Song, Jeannette Bohg, Szy- mon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Xiviv preprint rrXii:2305.05655 2023.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Safoora Yousefi, Leo Betthauser, Hosein Hasanbeig, Akanksha Saran, Raphaël Millière, and Ida Momennejad. In-context learning in large language models: A neuroscience-inspiree analysis of representations. arXiv preprint XrXe:22310.000 00313, 2023.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks aligned language models. arXiv preprint (tbb:2377.10433 2023.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Contents
Introduction
1
Related Works
3
3
Our Approach
4
3.1
Semantic Uncertainty Quantification
4
3.2
Vocal Affective Cue Analysis
5
3.3
In-Context Prompting
6
4
Scoring Choices by Polling LLMs
4
New Dataset: Disfluent Navigational Instruction Audio Dataset (DNIA)
6
5
Experimental Results
8
Evaluation Metric: Confidence measure
8
5.2
Confidence Score Improvement
Choice Distribution Change
Winning Rate
5.5
Ablation Study
10
6
Robustness to Adversarial LLM Attacks
10
Conclusions and Open Questions
11
A LLM Implementation Design
19
B Transcription Model Prompting
19
C Example Prompts and LLM Outputs
20
C.1 Prompts and Examples for Generating Reasoning and Choices (GPT-4
20
C.I.1 Without Affective Cue
20
C.1.2 With Affective Cue
22
C.2
Prompts and Examples for Choice Selection (GPT-3.5
23
C.2.1
Transcript Only
24
C.2.2
With Transcription and Reasoning
................................
25
C.2.3
Transcription, Reasoning, and Affective Cue
26
C.3
Large Language Model Adversarial Token Manipulation Attack Prompt
................................
27
D
Adversarial Attack Example
28
17

---

Published in Transactions on Machine Learning Research (10/2024)
E DNIA Dataset Details and Samples
28
F Pitch and Loudness Shifts Detection
29
G Sensitivity Study to Loudness and Pitch Threshold
29
User Interface for Human Evaluation on DNIA Dataset
30
18

---

Published in Transactions on Machine Learning Research (10/2024)
A LLM Implementation Design
We present the pipeline of querying LLM to analyze human audio inputs in Figure 0 GPT-4 is used for choice generation in the first stage, with in-context examples provided. The response contains the chain- of th-ohhghht reasoning process for the final action choice. The LLM output is cascaded GPT-3.5 for scoring the confidence of each choice. In each stage, we enforee LLM return output in JSON format, ensuring consistency in both our implementation and the parsing process. We also encapsulate the prompts in API request parallel processor lo speed up workflow. For clarily and transparency, used
In-Context Ex
Final Choice
System Prompt
Choice Generation
Multiple Choices
Scoring
(*D")
Audio
Transcription+ Affective Cue
GPT4
GPT3.5
Figure 6: Beyond Text Implementation Flow with LLMs. Final Choice 'D' only represents the output format. It output any choices from A,B.C,D,E.
gpt-4-1106-previ.e for choice generation and 1it-33.--turb--nnstte model. For Gemini-1 Manrini-1.5-Pree and 1pt-4-1106-previex, we used default hyperparameters except for the temperature, which set to 0. For gpt-3. Ept-3.5-turbo b-turbo-instrucc, text-davinci-003 we set max_tokent 100; temperature: echo: true; and the following hyper-parameters:
Table Logit Bias Settings
Token ID
Character (with space)
Logit Bias
362
A
100.0
426
B
100.0
356
C
100.0
423
D
100.0
469
E
100.0
B Transcription Model Prompting
We employ the state-of-the-art Whisper model RRdford et al. 2023) to transcribe the human audio-guided instruction. We use Whisper-Small with 244M paramelers that require 2GB VRAM to maintain the seamless transcription speed (within second). We add custom prompting 'Umm, let me think like, hmm... Okay, here's what I'm, like, thinking. enhance its proficiency capturing subtle nuances of speech, such word mumbling, hesitations, and speech repairs that may sound like slip of the tongue, these crucial uncertainty-related semantic signals.
19

---

Published in Transactions on Machine Learning Research (10/2024)
C Example Prompts and LLM Outputs
C.1 Prompts and Examples for Generating Reasoning and Choices (GPT-4)
You are a robot exploring building. You are in the middle of the building and are seeking directions from human.
Your gool is to provide possible neat sleps based on human response. A human response may not be trustworthy and contain disftuency.
You should figure out the unreliable part of the human response and generate five different neat-step actions based on the reasoning.
Your answer must strictly follow the JSON format the erample below provided. You must strictly Sollow the formal of the example provided below for your reasoning and nerl-slep actions.
Each next-step action must be distinct. The last nert-step action, E), must be *Ask another person near you for direction." For the rest of the next actions, if there is uncertainty, say "ask other people for further instruction' al the instruction where the uncertainly oppears.
Do not peraphrase. Always provide reasoning.
Prompt 1: System prompt for generating reasoning and choices.
C.1.1 Without Affective Cue
Task:
Find the direction to Cafeteria
Human Response: O # To be filled inc
Question: What possible next steps? Please always give me 5 possible answers, starting with A), B), C), D), E).
Prompt 2: Prompt for generating choices and reasoning without affective cue.
In-Context Example
20

---

Published in Transactions on Machine Learning Research (10/2024)
User Request
Task: Find the direction to the toilet
Human Response: Go straight, and err, take the second left
Question: What are possible next steps? Please always give me 5 possible answers, starting with A), B), C), D), E).
Agent Response
{
"Reasoning":" is hesitation signals that shows human reaponse is not trustworthy.",
"Answer":
"A": "Go straight, take the second left", "B": "Go backward, take the second left", "c": "Go straight, take the first left", "D": "Go atraight, the second left, ask another person for detailed guidance",
"E"; "Ask another person near you direction",
Example Input & Output Human Audio:
"So you start from the mail office and then you just walk straight and take the... maybe just take the first right turn and then you go straight and you will the cafeteria."
LLM Output:
"Reasoning": Thee use of 'maybe' indicates uncertainty in the human response, which makes not fully trustworthy."" "Answer":
Start from the mail office, walk straight and take the first right turn, then go straight and you will see the cafeteria
B: Start from the mail office, walk straight and take the second right turn, then go straight and you will see the cafeteria
C: Start from the mail office, walk straight and take the first left turn then go straight and you will
the cafeteria
D: Start from the mail office, walk straight and at the first right turn, ask another person for further instruction
E: Ask another person near you for direction
}
21

---

Published in Transactions on Machine Learning Research (10/2024)
C.1.2 With Affective Cue
Task:
Find the direction Lab
Human Response: {} # To be filled in
Human Vocal Affective Cue: ( # To be filled in
Question: What are possible next steps? Please always give me 5 possible answers, starting with A), B), C)2 D), E).
Prompt 3: Prompt for generating choices and reasoning with affective cue
In- mn-teotext Example
User Request
Task: Find the direction to the toilet
Human Response: Go straight, and err, take the second left
Human Vocal Affective Cue:
Large loudness decrease: No Change
Large pitch change: time = 02.200 second
Duration: "Go straight." 100.000, 00.360] (0.36 seconds);
Duralion: "and err, lake the second left. (00.360, 03.480] (3. 12 seconds).
Question: What are possible next steps? Please always give me possible onswers, starting with A), B), C)2 D), E).
Agent Response
"Reasoning":" is a hesitation signals that shows human reaponse is not trustworthy. The long duration on 'and err, take the second left 3.12 seconds, shows a lack of certainty on whether to take second left arrive at toilet. The large pitch is sign of intonation increase that sounds like question mark. It is at 02.200 second, in the range [60.360, 03.480] (last instruction phrase), confirm that the last part of response, 'and err, take the second left', is not trustworthy.".
"Answer":
"A"; "Go straight, take the second left", "B"; "Go backward, take the second left",
"C"; "Go straight, take the first left",
"D": "Go straight, the second left, ask another person for detailed guidance",
"E": "Ask another person near you for direction",
Example Input & Output
22

---

And then you go straight and Published in Transactions on Machine Learning Research (10/2024)
Human Audio and Affective Cue:
"So first, when you start from lab A, you go straight and you will pass lab A and then the locker room. And then you will probably need make left you will see lab B."
Large loudness decrease: No Change Large pitch change: at time = 23. 000 second
Duration: "So first," [0.000, 160] (2.16 seconds);
Duration: "when you start from lab A," => 160160, 4. 200] (2.04
seconds);
Duration: "you wanna go straight" => [4.200, 6.8401 (2. 64 seconds);
Duration: "and you will pass A" 840 9.280] (2.44 seconds);
Duration: "and then the locker room." => [9. 280, 12.440) (3.16
seconds);
Duration: " "And then you will probably need to make left turn." " => [12.440, 19.8 840] (7.40 seconds);
Duration: "And then you go straight" => [19. 840, 22.720] (2.88 seconds);
Duration: "and you will lab B. " [22.720, 28.080] (5. 36 seconds ););
LLM Output:
{
"Reasoning"; "The phrase "And then you will probably need to make a left turn" has long duration of 7.40 seconds, which indicates hesitation or uncertainty change at 23. 000 second, which falls within the duration of "and you will see lab B", suggests that the speaker is unsure about the final part of the instruction. The use of the word 'probably' also adds to the uncertainty of the direction to make a left turn.",
the direction given. Additionally, the large pitch
" "Answer":
A: Start from lab A, straight, pass lab and the locker room, make left turn, go straight, and you will
lab
B: Start from lab A, go straight, pass lab A and the locker roon, and ask other people for further instruction
C: Start from lab A, go straight, pass lab A and the locker roon, make right turn, go straight, and you will see lab
D: Start from lab A, go straight, pass lab and the locker roon, make left turn, and then ask other people for further instruction
E: Ask another person near you for direction
}
C.2 Prompts and Examples for Choice Selection (GPT-3.5)
Here, we show the success results for a vocal uncertainty(VU) example. Without Beyond Tezt, the LLM picks choice A, blindly following the human instruction. With reasoning, the same result. With Beyond Tezt, the LLM changes its choice from A B, successfully identifying the uncertainty within the human voice.
23

---

Published in Transactions on Machine Learning Research (10/2024)
You robot exploring building. You in the middle of the building and seeking directions from human.
Your gool is to provide possible next steps based on human response. The human response may not trustworthy and contain disfluency. Based on the following information, find the optimal choice that most accurately reflects the human response.
If the human response is not trustworthy, you should choose the choice that asks for help at the uncertain instruction part.
Prompt 4: System Prompt for selecting an answer from choices.
C.2.1 Transcript Only
Human Response: O # To be filled in
Your goal is to find the direction to the Cafeleria
Question: where should go next?
Choices: 0 # To be filled in
Answer:
Prompt 5: Prompt for selecting answer from choices with transcription only.
Example Input & Output
Human Audio:
"So start from the mail office, you wanna go straight and then when you are at the auditorium, you make a left turn. And then you go straight again and here you wanna take the first right turn and then if you just go straight again, you will see the cafeteria."
Choices:
A: Start from the mail office, go straight, at the auditorium make left turn, go straight, take the first right turn, then go straight again until you see the cafeteria
B: Start from the mail office, go straight, at the auditorium make a left turn, go straight, take the first right turn, then ask other people for further instruction
C: Start from the mail office, go straight, the auditorium make right turn, straight, take the first right turn, then straight again until you see the cafeteria
D:
Start from the mail office, go straight, at the auditorium make a left turn, go straight, take the second right turn, then go straight again until you see the cafeteria
E: Ask another person near you for direction
LLM Output:
24

---

Published in Transactions on Machine Learning Research (10/2024)
Start from the mail office, go straight, at the auditorium make a left turn, go straight, take the first right turn, then go straight again until you see the cafeteria
C.2.2 With Transcription and Reasoning
Human Response: 0 # To be filled in
Your goal is to find the direction to the Cafeteria
Reasoning is: 0 # To be filled in
Question: where should you go neat?
Choices: 0 # To be filled in
Answer:
Prompt 6: Prompt for selecting answers from choices with transcription and reasoning.
Example Input & Output
Human Audio:
"So start from the mail office, wanna go straight and then when you are at the auditorium, you make a left turn. And then you go straight again and here you wanna take the first right turn and then if you just go straight again, you will see the cafeteria."
Reasoning:
The human response contains filler words like 'So' and 'just', which typically do not alter the directions provided. The directions seem clear and without any contradictory statements.
Choices:
A: Start from the mail office, straight, at the auditorium make left turn, go straight, take the first right turn, then go straight again until you see the cafeteria
B: Start from turn, straight, take the first right turn, then ask other people for further instruction
mail office, go straight,
the auditorium make left
C: Start from the mail office, go straight, at the auditorium make right turn, go straight, take the first right turn, then go straight again until you
the cafeteria
D: Start from the mail office, go straight, at the auditorium make left turn, go straight, take the second right turn, then go straight again until you see the cafeteria
E: Ask another person near you for direction
25

---

Published in Transactions on Machine Learning Research (10/2024)
LLM Output:
A: Start from the mail office, go straight, at the auditorium, make left turn, go straight, take the first right turn, then go straight again until you the cafeteria
C.2.3 Transcription, Reasoning, and Affective Cue
Human Response: O # To be filled in
Human Affective Cue: 0 To be filled in
Your goal is to find the direction to the Cafeleria
Reasoning is: 0 # To be filled in
Question: where should you go next?
Choices:
0 # To be filled in
Answer:
Prompt 7: Prompt for selecting an answer from choices with transcription, reasoning, and affective cue.
Example Output.
Human Audio:
"So start from the mail office, you wanna go straight and then when you are at the auditorium, you make a left turn. And then you go straight again and here you wanna take the first right turn and then if you just go straight again, you will see the cafeteria."
Human Affective Cue:
Large loudness decrease: time 17.000 second Large pitch change: at time 23. 000 second
Duration: start from mail office, " );
[o. 000, 4.880] (4.88 seconds
Duration: "you vanna go straight"
[4.880, 8 680] (3. seconds);
Duration: "and then when you are at the auditorium," => [8 680, 11. 11.040]
(2.36 seconds);
Duration: "you make left turn." => 040, 12.480] (1.44 seconds); Duration: "And then you go straight again" [12.480, 15. 680] (3. 20
seconds);
Duration: "and here you vanna take the first right turn" => [15.680. 680, 260560] (6.88 seconds);
Duration: "and then if you just straight again,, => [22 560, 28. 600]
(6. 04 seconds);
26

---

Published in Transactions on Machine Learning Research (10/2024)
Duration:
"you will the cafeteria." [28.600, 34. 640] (6. 04 seconds)
Reasoning:
The human response contains large loudness decrease at 17.000 second and large pitch change at 200 second, which correspond to the phrases "and here you wanna take the first right turn" and "and then if you just go straight again," respectively. These vocal affective cues suggest uncertainty in these parts of the instructions. The duration of the phrases "and here you wanna take the first right turn" (6.88 seconds) and "you will longer than the other parts, which could indicate a lack of confidence certainty these directions.
the cafeteria. " (6.04 seconds)
also
Choices:
A: Start from the mail office, go straight, at the auditorium make left turn, go straight, take the first right turn, then go straight again until you see the cafeteria
B: Start from the mail office, go straight, the auditorium make left turn, straight, take the first right turn, then ask other people for further instruction
C: Start from the mail office, go straight, at the auditorium make right turn, go straight, take the first right turn, then go straight again until you see the cafeteria
D: Start from the mail office, go straight, the auditorium make left turn, straight, take the second right turn, then go straight again until you see the cafeteria
E: Ask another person near you for direction
LLM Output:
B: Start from the mail office, go straight, at the auditorium make left turn, go straight, take the first right turn, then ask other people for further instruction
C.3 Large Language Model Adversarial Token Manipulation Attack Prompt
The LLM atlack uses GPT4 as the adversarial paraphraser to attack the original text input. The allack takes the sub-instruction list of each original audio clip as input and then paraphrases it to reverse the semantic tone. The GPT4 attack prompt is shown below:
You are a paraphvaser. Your goal to change the level of certainty of the human instruction from an audio tronscript.
You achieve your goal by removing all uncertain cues in the text, such 'probably', 'maybe', 'possibly'' 'should' and make the semantic of the transcript very confident certain by adding words like 'certainly' ' 'must', or 'without doubt'.
You will take a transcript formatted as a list of timestamped sentences and produce a paraphrase of the transcript, strictly following the format of the original transcript.
Prompt 8: Prompt for adversarial token manipulation attack.
27

---

Published in Transactions on Machine Learning Research (10/2024)
D
Adversarial Attack Example
Before Attack:
"So first, when you start from lab A, you wanna straight and you will pass and then the locker room. And then you will probably need to make left turn. And then you go straight and you will see lab B."
Large loudness decrease: No Change 23.000 second
Large pitch change:
Duration: "So first," => [0.000, 2. 2.160] (2.16 seconds);
Duration: "when you start from lab A," [2.160, 4.200] (2.04
seconds);
Duration: "you wanna go straight" => [4.200, 6. 8.840] (2. 64 seconds);
Duration: "and you will pass lab A" => [6.840, 9. 9.280] (2. 44 seconds);
Duration: "and then the locker room." => [9.280, 14.4 (3.16
seconds);
Duration: "And then you will probably need to make a left turn." => 19. 840] (7.40 seconds);
Duration: "And then you straight" => [19. 840, 22.720] (2.88
seconds);
Duration: "and you will see lab B." => [22. 720, 28. 080] (6. (6.36 seconds
););
After Atlack:
"First, start from lab A, you straight and pass lab A and then the locker room. And then you will make a left turn. And then you go lab B."
straight and you will
Large loudness decrease: No Change
Large pitch change: at time = 23.000 second
Duration: "First," [0.000, 2.160] (2.16 seconds);
Duration: "start from lab A." => 160 4.200] (2.04 seconds);
Duration: "you go straight" => [4.200, 6.840] (2.64 seconds);
Duration: "and pass lab A" => [6.840, 9.280] (2.45 44 seconds);
Duration: "and then the locker room." => [9.280, 12.440] (3.16
seconds);
Duration: "And then you will make a left turn." => [12. 440, 19.840] (7.40 seconds);
Duration: "And then you straight" => [19. 840 22.720] (2.85 88 seconds);
Duration: "and you will see lab B." "> (22.720, 28. 080] (5. 36 seconds
););
E DNIA Dataset Details and Samples
For VU clips, we ensure no semantic uncertainty signals, such as uncertain-related words (i.e., 'maybe,' 'I guess'), speech repairs, and hesitations. We believe those are common affective characteristics associated with human speoch. However, those affective cues are not prominent LU clips. As our position states that affective cues (duration, pitch, loudness) are associated with speech uncertainty levels in robot navigation fasks, believe the setup of our dataset comprehensive enough to show that vocal cues good predictors.
28

---

Published in Transactions on Machine Learning Research (10/2024)
DNIA dataset is in GitHub repo: lattps://github. httpa/////ooooooaaaa///11AAAA--- come with textual annotations (grounded labels) well as Whisperr-generated transcription to save resources for other researchers who can reproduce the result.
The dataset will
Details on human choices for 500 DNIA audio clips are as follows: The user-study website is shown in Figure
Table 7: Human Choices Distribution
A
B
C
D
E
Total
66
154
95
96
89
LU
40
90
52
53
50
VU
26
64
43
43
39
The instructions emphasized careful listening and unbiased selection, with constraint of one minute per clip lo simulate real-world decision-makingg time pressures. We recruited diverse group of 20 participanis, varying in gender and dialect backgrounds, to ensure a broad range of perspectives.
From the dataset's ground truth scores, it's evident that the majority of correct choices fall within the 'B,, "D' options. This aligns with the results Figure | further demonstrating that our method achieves higher accuracy than the baselines.
F Pitch and Loudness Shifts Detection
Algorithm ! detects the largest shifts in pitch and loudness in a given audio clip. We use the timestamp to align with the sequence of instructions for vocal cue prompts.
Algorithm Loudness and Pitch Detection
Require: audio V, 02 (loudness threshold), Op (pitch threshold)
Ensure: Timestamps of maximum loudness change (4x) and pitch shift (4p) 0,
1: -, P
Loudness and Pitch changes
Iprev:Ppue Pprev < loudness and pitch at [to, [1]
for each second in V do
Define segment as t++1] he- avg loudness (segmeet))
6: Phat + calculate_ pptpppichhhhhc (segment)
7:
14a11 - -pprevl > 0, then
8: Append (t++1, 14... - (prevv) to L
end if
10: if |P..: - Pprevl > 0p then
11: Append (lit1, |Ptes Pprev 1) to P
12: end if
13: 1prev & Lem Pprev t Pt.-i
14: end for
15: f + argmax, (ōr, (t, d1,) E L}
16: 1p f- argmax, (6p, (1, 8p,) E P]
G Sensitivity Study to Loudness and Pitch Threshold
We conducted threshold experiment on loudness and pitch to demonstrate the effectiveness of our method. With loudness levels set at 0.03 and 0.05, and pitch set at 300 Hz and 500 Hz respectively, the winning rale of our method consistently outperformed the baseline methods by a significant margin. Specifically, compared to the 'Transcription Only LLM," which had winning rate of 22.16% (total), 21.75% (Language Uncertainty, LU), and 22.79% (Vocal Uncertainty, VU), and the 'With Chain-of Thought Reasoning' method, which achieved 49.3 30%30% (total), 58.60% (Language Uncertainty, LU), and 36.5 74.7 (Vocal Uncertainty, VU),
29

---

Published in Transactions on Machine Learning Research (10/2024)
method showed superior performance (These baseline scores are also shown in Table 3 While we acknowledge that the robustness may not. be consistent across all scenarios, the results indicate that the proposed met hod can be effectively generalized to diverse environments.
Table 8: Sensitivity Study
Condition
Total
LU
VC
Loudness 0.03, Pitch 300 Hz
0.704
0.688
0.726
Loudness 0.05, Pitch 300 Hz
0.614
0.600
0.633
Loudness 0.03, Pitch 500 Hz
0.616
0.593
0.647
Loudness 0.05, Pitch 500 Hz
0.602
0.596
0.609
H User Interface for Human Evaluation on DNIA Dataset
This section details the user interface that used for human assessment. As shown in Figure 7 the website allows users to log in and participate in the human evaluation session. The user is asked to listen to randomly selected audio clip from the audio dataset and answer the question by choosing the most accurate and efficient answer from list of five options. Once the user select the answer, they need to hit Confirm button to submit the answer the proceed to the next question.
DMA Cerseet Usar Annotation
Figure 7: Screen capture our DNIA dataset human annotation website.
30


Reviews:
[{"review_id": "8fd6cGLNEQ", "reviewer": "Reviewer_odF2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper effectively argues that existing text-based LLMs fall short in interpreting the nuances of human language, particularly in social navigation tasks where vocal cues play a crucial role in conveying uncertainty.  To address this, the paper introduces Beyond Text, a framework that incorporates both audio transcription and affective vocal features (pitch, loudness, duration) to improve robot navigation, particularly in situations where instructions are ambiguous. \n\n2. To support this approach, the paper introduces the Disfluent Navigational Instruction Audio Dataset (DNIA), addressing the gap in existing datasets by providing audio clips capturing vocal and textual uncertainties characteristic of real-life human speech in navigational contexts.\n\n3. The paper is well-presented and the experiments are comprehensive, demonstrating the effectiveness of the proposed method (I'm not an expert in this specific domain).\n\nWeaknesses:\n\nThe main weakness is that the proposed method is mainly based on heuristics and there is no theoretical analysis. Moreover, the paper follows a strong assumption that the duration, pitch, loudness reflect the confidence of the speaker. This assumption might not hold in other cases. Therefore, it might be hard to valid its effectiveness in other datasets or changing environments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This research paper investigates the use of vocal cues to improve LLMs for robot navigation tasks. The authors argue that current methods struggle with interpreting ambiguous and uncertain instructions given through audio, often missing the nuances conveyed in a person's voice. To address this, they propose a new approach called \"Beyond Text\" which integrates audio transcription with an analysis of affective vocal cues like pitch, loudness, and duration. This allows the LLM to better understand the confidence and uncertainty in human instructions, leading to more accurate decision-making for robots. They also introduce a new dataset called Disfluent Navigational Instruction Audio Dataset (DNIA) specifically designed to capture vocal and textual uncertainties present in real-life navigational guidance. Through experiments, they demonstrate that Beyond Text significantly outperforms existing LLM-based approaches for human audio-guided navigation, showing robustness against adversarial attacks and showcasing the potential of audio augmentation for LLMs in human-robot interaction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Required:\n\n1. It is suggested to conduct sensitivity study to the parameters of loudness threshold and pitch threshold in Algorithm 1.\n\nSuggested:\n\n2. Suppose one uses an audio-to-text tool to convert the audio to text first and then uses a text-to-audio tool to convert the text to audio. Then they input the generated audio to the proposed method. It would be interesting to see how the proposed method works in this case. Will the proposed method still show performance gain?", "broader_impact_concerns": "A Broader Impact Statement is needed for this paper. For example, potential biases in interpreting vocal cues (like those stemming from gender, accent, or cultural background) need to be carefully discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ts8gMYjPpV", "reviewer": "Reviewer_FxnN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally clear on the implementation, I appreciate having direct transcripts of the prompts and responses because it was not actually that clear from the maintext how this was done. The ablations between chain-of-thought alone vs chain-of-thought + audio cues is appreciated.\n\nThe paper not using any direct multimodal LLM that natively takes audio does harm some of the conclusions (although this is mentioned in the conclusions as an open question).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors note that LLMs have shown promise as decision-making tools for robot navigation, but pure text can drop relevant information from human instructions. Assuming a context where people speak to robots, the proposed method is to use audio cues rather than just the transcribed audio. An example of an audio cue is the duration spent on each sentence, to check if the speaker is speaking quickly or slowly.\n\nThe focus on the audio cues is to determine locations where the person providing instruction may be uncertain. To use these audio cues, the LLM does not take audio natively - instead it takes a summary of audio cues as additional part of text. This includes large changes in volume, larges changes in pitch, and the time spent saying each sentence of text. Transcription is done using Whisper, slightly prompted to do better at parsing mumbled text.\n\nThe resulting method is found to lead to better success rates on two datasets constructed to have vocal uncertainty (VU) and language uncertainty (LU), with largest gains on the VU dataset, indicating vocal cues can be helpful.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would appreciate a clearer definition of winning rate - winning rate relative to what? How many data points is the comparison over?", "broader_impact_concerns": "It is a more minor point but I do wonder if there are potential fairness concerns for people who are less able to vocally express themselves, tend to speak with more vocal uncertainty, etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ODEtneAl5P", "reviewer": "Reviewer_cPuV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper shows strong improvements for the win rate. \n- I appreciated the detailed examples in the appendix.\n\n### Weaknesses\n\n**Confidence Measure**\n\nIn 5.1, the $\\rho$ distribution is not really defined, is it the probability of each possible candidate under $p_\\theta$ divided by the sum of $p_\\theta$ for all the candidates? This distribution is very sensitive to the possible candidates and not obviously interpretable in between set of candidates. \n\nThe quantity in (4) is not a KL divergence. In addition, a true KL between between $\\rho$ and $\\rho^*$ is not defined as it exists $j$ in $J$ such that $\\rho(j) \\neq 0$ and $\\rho^*(j) = 0$. The KL that exists is KL($\\rho^*$,$\\rho$).\n\nFrom the second sentence of 5.2, the confidence score could be KL($\\rho^*$,$\\rho$). Is that the case? If yes, I do not really understand why it would be called a ‘confidence measure’ or a ‘variance’ as it does not take into account the whole distribution but just the probability weight on the ground truth possibility.\n\n**Improvement on the LU subset with VU information**\n\nIn Table 2, on the LU subset, the scores improve with the pitch/loudness/duration information. Would it mean that there is a leak of VU information on the LU subset? \n\n**Dataset**\n\nLack of some information in the dataset:\n  - The distribution of scores (i.e how many A,B,C,D,E as this as a large impact on the winning rate of the baseline)\n  - The inter rater agreement to have a better understanding of the upper bound\n  - As TMLR does not have a deadline why not waiting to have the dataset released before the submission as the dataset is claimed as a contribution? \n\n\n**Question about the setup**\n\nWhy not simply disambiguate the instruction explicitly instead of implicitly taking into account the uncertainty? Disambiguating the instruction explicitly would simply require asking an additional question to make sure the human is uncertain.\n\nNot clear to me if we want the robots to infer the real instructions and not just follow the instructions given to the robot. The disfluencies can be due to external factors (e.g. distraction from someone) and have nothing to do with the actual uncertainty of the human interacting with the robot.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to better use the uncertainty expressed in audio. A given audio is translated into text and an effective analysis model provides additional information about the vocal cues like the duration, the pitch change or the loudness.  This additional information is injected into the prompt. The results show that reasoning and reasoning + the proposed prompt augmentation based on an effective analysis model help choosing the right next-step action. Additionally, the authors say that they plan to release a dataset called Disfluent Human Audio-Guided Instructions which corresponds in 500 human audio navigational instructions with perceptual “ground truth human labeling”. Finally, the authors ran an ablation study to show that their approach is robust to adversarial attacks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Confidence Measure**\n\nMore information on how the confidence measure is computed is critical as there are some inconsistencies in equation (4). \n\n**More information on the dataset**\n\nMore information about the dataset is important:\n  - The distribution of scores (i.e how many A,B,C,D,E as this as a large impact on the winning rate of the baseline)\n  - The inter rater agreement to have a better understanding of the upper bound\n  - Full release of the dataset as it is mentioned as a contribution\n\n**False negative rate**\n\nIt could be interesting to compute the false negative rate i.e. the amount of time the choice A was not chosen while it was indeed the intended choice. The negative rate could be especially interesting in the case of LU only. \n\n**Minor changes**\n\nEspecially in the Related Works, \\citep should be used instead of \\citet in many places.", "broader_impact_concerns": "I do not foresee any ethical implication of the work that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were all satisfied with the experiments and ablations conducted in the paper to support the need for vocal cues as part of the LLM for ambiguous navigation tasks. The reviewers also appreciated the release of the supporting dataset.

Most of the reviewer-author discussion had to do with a few additional experiments and some clarifications (e.g., metric definitions). In the end, all reviewers were happy with the paper for acceptance to TMLR.
Decision: Accept (poster)