Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Authors: Allen Z. Ren
Anushri Dixit
Alexandra Bodrova
Sumeet Singh
Stephen Tu
Noah Brown
Peng Xu
Leila Takayama
Fei Xia
Jake Varley
Zhenjia Xu
Dorsa Sadigh
Andy Zeng
Anirudha Majumdar
Venue: CoRL 2023 Oral (2023)
Keywords: Language-based planning
uncertainty estimation
conformal prediction

Abstract: Large language models (LLMs) exhibit a wide range of promising capabilities --- from step-by-step planning to commonsense reasoning --- that may provide utility for robots, but remain prone to confidently hallucinated predictions. In this work, we present KnowNo, a framework for measuring and aligning the uncertainty of LLM-based planners, such that they know when they don't know, and ask for help when needed. KnowNo builds on the theory of conformal prediction to provide statistical guarantees on task completion while minimizing human help in complex multi-step planning settings. Experiments across a variety of simulated and real robot setups that involve tasks with different modes of ambiguity (for example, from spatial to numeric uncertainties, from human preferences to Winograd schemas) show that KnowNo performs favorably over modern baselines (which may involve ensembles or extensive prompt tuning) in terms of improving efficiency and autonomy, while providing formal assurances. KnowNo can be used with LLMs out-of-the-box without model-finetuning, and suggests a promising lightweight approach to modeling uncertainty that can complement and scale with the growing capabilities of foundation models.

Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Allen Z. Ren Anushri Dixit', Alexandra Bodrova', Sumeet Singh", Stephen Tu², Jake Varley", Zhenjia Xu', Dorsa Sadigh", Andy Zeng", Anirudha Majumdar Princeton University, "Googe DeepMind
Noah Brown", Peng Xu², , Leila Takayama', Fei Xia?, Jake
Large language models (LLMs) exhibit a wide range of promising capa- bilities - from step-by-step planning to commonsense reasoning - that may provide utility for robots, but remain prone to confidently hallucinated predictions. In this work, we present KNOwNo, framework for measuring and aligning the uncertainty of LLM-based planners, such that they know when they don't know, and ask for help when needed. KNOWNo builds on the theory of conformal prediction to provide statistical guarantees on task completion while minimizing human help in complex multi-step planning settings. Experiments across a variety of simulated and real robot setups that involve tasks with different modes of ambiguity (c.g., from spatial to numeric uncertainties, from human preferences Winograd schemas) show that KNowNo performs favorably over modern baselines (which may involve ensembles extensive prompt tuning) in terms of improving efficiency and autonomy, while providing formal assurances. KNOWNo can be used with LLMs out-of-the-boo without model-finetuning. and suggests promising lightweight approach to modeling uncertainty that can complement and scale with the growing capabilities of foundation models.'
Abstract:
Keywords: Language-baser planning, uncertainty estimation, conformal prediction
Introduction
How can we endow our robots with the ability to know when they don't know? Accurately modeling and accounting for uncertainty is longstanding challenge towards robots that operate reliably in unstructured and novel environments. In this work, study this challenge in the context of language inntrructe robots. Language provides natural and flexible interface for humans to specify tasks, contextual information, and intentions, while also allowing us to provide help and clarification to robots when they are uncertain.
Recently, approaches that leverage large language models (LLMs) for planning [1, 2] have demonstrated the ability to respond to natural and unstructured language instructions to generate temporally extended plans. These approaches enable leveraging the vast amount of prior knowledge and rich context embedded in pretrained L.LMs, and lead to substantial abstract reasoning capabilities. However, one of the major challenges with current LLMs is their tendency to hallucinate, i.e., to confidently generate outputs that are plausible but incorrect and untethered from reality. Such false confidence in incorrect outputs poses significant challenge MLLM-based planning in robotics. Natural language instructions in real-world environments often contain high degree of inherent ambiguity, and confidently following an incorrectly constructed plan could lead to undesirable or even unsafe actions.
As example, robot tasked with heating food may choose place metal bowl in the microwave, potentially damaging the microwave or even causing fire (Fig. 1). Instead of following such plan, we would like our robots to know when they don't 't know and ask for clarification instead (e.g.. ask which bowl is safe to be placed in the microwave). Prior work in language-based planning either does not seek such clarifications [1] or does so via extensive prompting [2], which requires careful prompt engineering to prevent the robot from excessively relying on seeking assistance. Moreover, prior approaches do not provide way to ensure that asking for help results in desired level of task success. We formalize
'Webpage with additional results and videos: httpp://robot-baet http:////oooot-allpgiitb....ib gii.hbb.ittbioo
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Robot Planning Human Interaction
Uncertainty Allgnment with KnowNo
Robot Environments
Hume
Place the microwave, please
There is a microwave, a landfil bin, a recycling bin, and compost bin
Observations: see a metal bowl and a plastic bowl on the counter.
Possible next steps:
0.44 plastic bowl microwave,
0.41 Puut metal bowl in microwave.
0.03 Put metal bowl in landfill bin
0.08 plastic
recycling
futell
Conformal prediction quantile: 0.21
Which one, plastic or metal?
Steps with scores above quantile:
0.44 plastic microwave.
0.41 PPu metal bowl In microwave.
The plastic one, please.
Prediction size
help
Question: Which one, plastic metal?
Figure I: KNOWNO uses Conformal Prediction (CP) to align the uncertainty of LLM planners. Given language instructions, an LLM generates a set of possible next steps (with scores above a certain CP calibration quantile). there are more than one in the set, the robot asks for help, then readjusts accordingly. Experiments show that KNowNo works well with multiple embodiments, across a variety of ambiguous situations, in both simulated and real setups. these challenges via two desiderata: (i) calibrated confidence: the robot should seek sufficient help to ensure statistically guaranteed level of task success specified by the user, and (ii) minimal help: the robot should minimize the overall amount of help it seeks by narrowing down possible ambiguities in a task. We collectively refer to these sufficiency and minimality conditions as uncertainty alignment.
Statement of contributions. We propose KNOWNO- Knnnww When You Don't 't Know framework for aligning the uncertainty of LLM-based planners utilizing the theory of conformal prediction (CP) [3, 4]. We make the following contributions: (1) Given a language instruction, we utilize a pre-trained LLM with uncalibrated confidence to generate a set of possible actions for the robot to execute next. We demonstrate how to use CP to select a subset of these options, which allows the robot to decide an action to execute (if the subset is a singleton) or ask for help otherwise. (2) We prove theoretical guarantees on calibrated confidence in both single-step and multi-step planning problems: with user- defined confidence 1 - c over instructions, the robot performs all actions for a given instruction correctly by asking for help when necessary. CP also minimizes the average size of prediction sets, thus addressing the goal of minimal help. (3) We evaluate KNOWNO in both simulation and hardware with a suite of language-instructed manipulation tasks with various types of potential ambiguities (c.g.. based on spatial locations, numerical values, attributes of objects, and Winograd schemas). Experiments across multiple settings and embodiments validate the ability of KNOWNO provide statistically guaranteed levels of task success while significantly reducing the amount of help required to achieve a given success level by 10-15% as compared to baseline approaches.
2 Overview: Robots that Ask for Help
Language-based planners. Language model planners can generate step-by-step robot plans, where each step y is composed of variahle-length sequences of symbols (2..0)..... e.g., text tokens as input language-conditioned policy [1] (see Fig. 1), or robot code executed by an interpreter [5]. Pretrained autoregressive LLMs predict each step y. whose joint probability over tokens can be factorized as the product of conditional probabilities of next token prediction p(y) = -i=__ppe are interested in characterizing the uncertainty of next step prediction p(y)- The distribution of p(y) values (i.c., perplexity) remains highly sensitive to luriable-length k; hence p(y) on its own serves as a rather poor scoring function [6] particularly when steps in plan are expressed in natural language (our experiments in Section A8 also show that using p(y) directly for calibration leads to poor performance).
o;.... Here, we
Planning multiple-choice Q&A.. We can address this length bias with simple trick, whereby can prompt the LLM to generate (either with high-temperature sampling, or with another LLM instance)
2

---

Diverse Scenarios
MCQA
Conformal Prediction
Calbration
Mutiple Cho Gencration
There is a metal bowl and plastic
D:0.39
Hantoriam of Non Confirmity
bowl Put the bowl in the microwave.
A) Put the dirty sponge in landfill bin
Scores of True Labela
the apple compost
There is Cake, Sprite, and bottled water.. Can you get me a sods?
C) Put the apple in landfill bin
D) Put the dirty sponge compost
E 20%
An option not listed here
rice and multigrain chips... Put rice chips the drower
Which option is correct?
Test Data
Sokan Lood
Prediction Set
apple and dirty sponge.. is rotten. Can you dispase it?
0.20 B: 0.56 C. 0.12 0.0.8
E.0.04
A-B-20 0.05 G---12 0-0-08
Figure 2: KNOWNo formulates LLM planning as MCQA by first prompting LLM to generate plausible options, then asking it to predict the correct one. Using the next-token likelihoods collected with calibration datasct, CP provides the quantile value such that all options with score 21 21- are included in the prediction set in test sccnario. a set {e/) of candidate next steps (e.g.. "Put plastic bowl microwave", "Put metal bowl in microwave", etc., in Fig. 1), and then format the task of choosing among them as multiple-choice Q&A (MCQA). This climinates plans that the LLM considers unlikely and reduces the problem of next-step prediction down to single next-token prediction aligning with LLM log- likelihoodd loss functions and LLM training data (e.g.. MCQA datasets [7, 8). These probabilities can serve as normalized scores that can be used by various un- certainty quantification methods such as thresholding, ensemble methods, and others. In this work, we use these normalized scores within a conformal prediction (CP) framework. Specifically, CP uses a held-out cal- ibration set of example plans in different scenarios to generate a reduced prediction set of plans among {v/) (Fig. 2). The LLM certain if this prediction set a singleton, and triggers help from a human otherwise.
Robots that ask for help. In this work, we show that LLM planning - combined with CP for uncertainty estimation - can effectively enable robots to interact with an environment, and ask for help when needed. The environment can be formulated as partially observable Markov decision process (POMDP): at any given state 8t at time t, given user instruction E, the robot executes an action a according to policy π, then transitions
new state St+1- Our policy is composed of four parts (Fig. 1): 1. Multiple-choice generation: An LLM generates a diverse set of candidate plans labeled with 'A', 'B', *C", DD' and additional possible plan, 'E) an option not listed here', which is appended post-hoc. We denote the set of labels by } = ( A', BB', ''', the LLM with context x²', which is text that (1) summarizes the robot observation at each time step (e.g., using a vision-based object detector or an oracle; see Fig. 1) and (2) includes the user instruction. An augmented context H' is obtained by appending the LLM LLM-generated plans to the context x".
These plans are generated by prompting
2. Prediction set generation: We use CP to choose subset C(**) CV of candidate plans using the LLM's (uncalibrated) confidence ((a"), in each prediction y y given the context .t.
3. Human help: If the prediction set is non-singleton, the robot leverages help from a human (or any other supervisor agent, denoted function fx) to arrive at an unambiguous next step yu E C(**).
4. Low-level control: A low-level module yp converts the plan in y/H to an action a, =
Goal: uncertainty alignment. Often in real-world settings, language instructions can be ambiguous. e.g. "place the bowl in the microwave" does not specify that the human may prefer the plastic bowl (Fig. 1). Our goal in this work is to address uncertainty alignment: achieve a desired level of task success while minimizing human help. We formalize this by considering a joint distribution D over scenarios {= (=,,L.)). where is an environment (POMDP), f is (potentially ambiguous) language instruction, and g is goal (e.g., formulated as a subset of acceptable states in the POMDP and partially observable through 1). Importantly, we do not assume knowledge of DD except that we can sample finite-size dataset of i..d. scenarios from it. We formalize uncertainty alignment in our setting as (i) calibrated confidence: the robot's policy (with human help as described above) succeeds with user-specified probability 1 -- over new scenarios E~ D, and (ii) minimal help: the policy minimizes the number (C(-) of options presented to the human on average across scenarios E~
3 Calibrating LLM Confidence with Conformal Prediction
The MCQA setup above allows us to apply CP obtain calibrated confidence guarantees while (upproximately) minimizing help. We introduce CP below, and then present the different practical settings consider (possibly involving multiple planning steps and/or multiple correct plans per step).
3

---

3.1 Background: Conformal Prediction
For now, drop the timestep superseript and consider generic MCQA setup with pairs (Z.y) consisting of input ä and true label y. Suppose there is culibration set Z = { (T:,3:) )) of such pairs drawn i..dd
from an unknown distribution 'D over 2 = x y.. Now, given a new i..dd sample Sest = (Frss -Mea)) with un- known true label Mhest CP generates a prediction set (TTea) yy' that contains yleat with high probability [3]:
IP [[ E ((nes) 2 1-c,
(I)
where 1-c is a user-defined threshold (desired task success level in our setting) that affects the size of C(-).
To generate C(mea), CP first uses the LLM's confidence f (cf. Section 2) to evaluate the set of nonconforiity scores {R² 1 - f(Ez)y, J==1 over the calibration set the higher the score is, the less each data in the calibration set conforms to the data used for training f. Then CP performs calibration by defining / to be the [[V+][[-=]]] empirical quantile of Ki.-.,NN. Lastly, CP generates N
((iee)== Ey (y 21 1- i.e., the prediction set that includes all labels that the predictor is least q confident in. The generated prediction set ensures that the coverage guarantee in Eq. (1) holds.
Dataset-conditional guarantee. The probability in Eq. (1) is over both the sampling of the calibration set Z and Rest (i.e., marginal guarantee). Thus, to ensure the desired probability of coverage for each new Ftest, one needs a fresh calibration set. Instead, we apply the following dataset-condition guarantee [9] which is conditioned on particular calibration dataset being sampled, and thus can be applied to new test data without re- callibbraiio:
PB E C(Tes)) (=I,...2N)) 2 Beta (6), v:= L(N 1))J;
it holds with probability 1 d over the sampling of Z. and Beta -1 (8) (quantile) level of 6 in a Beta distribution with parameters N -+1-u and v, and é is the threshold used for calibration. In practice, we use a modest-sized calibration dataset (N = 400) and ô= 0.01, and adjust è to achieve the desired 1 - coverage (with probability 1-8= 0.99 over the sampling of the calibration set).
denotes the inverse CDF
Minimal prediction size. From [10, Thm. IJ. C(-) achieves the smallest average set size among possible prediction schemes C. that achieve the coverage guarantee, if f() models true conditional probabilities:
min E CEC (a.
[[])]] subject to (1).
(3)
The assumption that / models true conditional probabilities may be good approximation for LLMs trained on large-scale data with proper scoring rule [11]: one can also obtain bounds on near-optimal average set sizes for CP using f that approximately model conditional probabilities [12, 10]. but we omit these results for brevity. Overall, CP is powerful and easy-to-ur statistical tool to produce (1) tight coverage guarantees addressing the goal of calibrated confidence and (2) small prediction sets for unseen data given a blackbox predictor like a LLM and an unknown data distribution aaciuuttioonnddeessnnnn our second goal of minimal help.
3.2 Single-Step Uncertainty Alignment
We now demonstrate how to use CP to achieve uncertainty alignment with a user-defined task completion rate 1-c. We first consider single-step setting, where the LLM plans only once given context. For simplicity, we again drop the timestep superscript t in this section.
Data collection. We collect N i.i.d. scenarios from the distribution D, and the corresponding contexts summarizing the robot observation and instruction. We use the MCQA approach from Section 2 to generate candidate plans and then label each augmented context ă (i.e., context combined with plans) with the correct label (here and in Section 3.3, we assume that there is unique correct candidate plan; we provide extension multiple acceptable options in Section A2). We thus obtain calibration set Z Z= (z+ ) with pairs of augmented contexts and correct labels.
Calibration. Next we follow Section 3.1 to perform calibration: first adjust é to achieve the 1 coverage based on Eq. (2) and then find the quantile level q. Given a new context 7iest (after MCQA in a new scenario) test time, we can construct the calibration set C(igas) that contains yees with probability.
Triggering help. If C(tea) is a singleton, the robot executes the corresponding plan. Otherwise, deem the LLM uncertain over possible actions and trigger human help. The robot presents the human

---

with (Trea) (including the corresponding plans in text) and asks the human to choose one. The human chooses Mest if thest E (irss)) or halts the operation otherwise. This setup turns the coverage guarantee from CP to the task completion guarantee:
Proposition 1 (Single-step uncertainty alignment) Consider a single-step setting where we use CP with coverage level 1-ctt to generate prediction sets and seek help whenever the set is not singleton. With probability 1 over the sampling of the calibration set, the task completion rate over new test scenarios drawn from D is at least c-c. If models true conditional probabilities, the average prediction set size is minimized among possible prediction schemes that achieve - completion rate.
The proof immediately follows from the fact that under the assumption of accurate human help. the robot fails only when the prediction set does not contain the true label; the prediction set minimality follows Eq. (3). Thus, our approach addresses the goals of calibrated confidence and minimal help from Section 2. 3.3 Multi-Step Uncertainty Alignment
Now we extend the CP casbd uncertainty alignment approach to settings where the LLM plans in multiple timesteps. This setting can be helpful when the L.M receives feedback from the environment or human between steps. However, the original CP formulation cannot be applied here since the context x" between steps are dependent; moreover, the robot's actions at step t influence the distribution over contexts that the robot observes at future steps. Thus, the i..d. assumption for the coverage guarantee is no longer valid. Here, we present novel extension of CP to multi-step settings that tackles this challenge.
Sequence-level calibration. The key ideas are to (i) lift the data to sequences, and (ii) perform calibration at the sequence level using a carefully designed nonconformity score function (that allows for causal construction of the prediction set at test time). Suppose that each data point consists of a sequence of augmented context
(7", ,,,", "..,....""""" 11) and true labels ÿ = (y/ .y' ....."" ....,y"''), 1), where T' is the time horizon and 2 arises from having performed the correct actions in previous steps. The distribution DD over scenarios induces a distribution over data sequences. We can again collect calibration set Z = [2+ = (I;.;:)) Next we use the lowest score over the timesteps as the score for the sequence':
} ( min fx""))
:E[T]
With the standard calibration procedure in Section 3.1, we construct a sequence-level prediction set (= (VEY") (T 2 1- for a new context sequence Trest with the quantile level q.
Causal construction of C( at test time. Note that C( (wex) is constructed with the full se- quence Tist at once. However, at test time, we do not see the entire sequence of contexts all at once but rather Tyent one at a time. We thus need to construct C (nstt) in a causal manner (i.c., always relying only on current/past information). Consider the causally constructed prediction set 4) time /. using the same quantile level ğ from the non-causal
C' (rics))) = {v' ((ffea)y calibration above, and define ((Tes) == (x C'l (IrLe)) X CT CTT'(x"())) We would like to obtain lower bound on the sequence-level coverage, P( E (((Tea)) 21-
Claim 1 For any yEYT, E (I = y E (1tear)
Proposition 2 (Multi-step uncertainty alignment) Consider a multi-step setting where we CP with coverage level 1 c to causally construct the prediction set and seek help whenever the set is not a singleton at each timestep. With probability 1- 5 over the sampling of the calibration set, the task completion rate over new test scenarios drawn from D is at least 1-e. If models true conditional probabilities, the avenuge prediction set size is minimized among possible prediction schemes that achieve 1 completion rate.
The proofs are deferred to Section Al. Claim allows us to construct causal prediction sets from non-causal calibration, and then we show the sequence-level task completion rate guarantee still holds.
Multiple acceptable options. Often, there can be multiple acceptable options the same timestep. e.g.. the task is to bring the human soda, and either the Coke Sprite on the table is acceptable. In such settings,
2If the correct option in C(res) is 'E', the human provides the correct action that was not listed by the robot. The robot continues with this action.
We overload notation here and use j to also assign confidence scores sequences.
5

---

we would like the prediction set to contain at least one acceptable option. We extend our method and confidence guarantees to this setting for both single- and multi-step problems in Section A2 and Section A3.
4
Experiments
We evaluate our framework in a diverse set of language-insirutted tasks and environments below, and demonstrate its effectiveness in achieving a user-specified task completion rate while minimizing user help. We use PaLM-2L [13] as the LLM in all examples unless otherwise noted. We parameterize the scenario distribution for each experiment with details shown in Section A6.
Baselines. A straightforward way construct prediction sets given desired 1 coverage is to rank options according to confidence and construct set such that the cumulative confidence exceeds 1-e; consider two baselines that are based on such cumulative thresholding but use different kinds of scores: (I)
Simple Set uses the same f as KNOwNo; (2) Ensemble Set [14] instead uses the frequencies of the LLM outputting y E y (out of 20 trials total) with randomized prompting. However, such ways of constructing prediction sets are not guaranteed to achieve 1 coverage as the probabilitics can be miscalibrated [15], and often include additional unnecessary options [16]. Instead of using cumulative thresholding, KNowNo constructs prediction sets by including options with scores higher than a threshold computed using CP. which results in statistical guarantees. We also introduce two prompt-hased baselines: Prompt Set prompts the LLM to directly output the prediction set (e.g.. "Prediction set: [A, CJ"); Binary prompts the LLM to directly output binary indicator for uncertainty (e.g., "Certain/ Uncertain:: Certain"), which is used in other LLM-based planning work [2] for triggering human intervention. Note that the level is not used in Prompt Set or Binary, and so the user cannot explicitly control the task success rate. Lastly, we consider No Help where the option with the highest score is always executed without any human intervention.
4.1 Simulation: Tabletop Rearrangement
0.6
KnowNo
Simple Set
A robot arm is asked to rearrange objects on table in the PyBulle simulator [17] (Fig. right middle). Each scenario is initialized with three bowls and blocks of green, yellow, and blue colors. The task is to move certain number of blocks or bowls towards different object or a specific location around it. We introduce three settings based different types of ambiguities in the user instruction: (1) Attribute (e.g..
0.5
Ensemble Set
Deviation from 1
Prompt Set
0.3
Binary
No Help
0.2
0.1
referring to the bowl with the word "receptacle"), ) (2) Numeric (c.g., under-specifying the number of blocks to be moved by saying "a few blocks") ) and (3) Spatial (e.g.. "put the yellow block next to the green bowl", but the human has a preference over placing it at the fron/bbacklefttrighh))
Attribute Numeric Spatial
Figure 3: Deviation from specified task success
level 1 - c 0.85 to the empirical success rate for the three settings in Simulation. 200 trials are run for each method/setting.
KNOWNo achieves target task success rate consistently.
First, we investigate whether KNOWNO and the baselines achieve a given target task success rate consistently the three settings - we set the failure level e =00.15. In Fig. 3 we show the difference between achieved and target rates for all methods. Results show that KNOWNo provides task success rates closest the target overall, due to the coverage guarantee from CP. Prompt Set, Binary, and No Help have larger deviations from the target since the user has no control over the envor rate. Also, the scenarios get increasingly ambiguous (least in Auribute and most in Spatial), the baselines show larger deviations.
KNOWNo achieves high task success rate with lower human help as varies. In Fig. 4 we vary the target error rate and show the full curves of task success rate vs. prediction set size and human help rate averaged over the three settings. For KNOWNo. Simple Set, and Ensemble Set, specifying a lower improves the empirical task success rate while also requiring more human intervention. The most natural comparison is between KNOwNo and Simple Set; both use next-token probabilities from the LLM as the confidence score but KNOWNo applics CP and Simple Set applies cumulative thresholding. KNOWNo achieves higher success-to-hell ratios across 6 levels, thanks to calibrated confidence from CP. Meanwhile, Prompt Set and Binary do not allow controlling success rates. Prompt Set performs the worst, indicating the challenge of prompting based methods for calibrated prediction set. Binary performs favorably at certain success levels but is (1) inflexible and (2) does not provide human with prediction set for casier feedback.

---

In addition, Fig. A10 shows the results for individual ambiguity settings. As the scenarios become more ambiguous, KNOWNO shows a greater reduction of human help compared to Simple Set- as much as 24% at certain success levels.
1.0
1.0
Task success rate
Ensemble Set can perform well
0.9
0.9
but is computationally expensive. Fig. 4 also shows that Ensemble Set provides high task success rate with small amount of human help higher levels. However, there are two main drawbacks. First, we find in some sce- narios that even with 20 randomized prompts, the L.M can fail to choose the correct option and thus assigns zero probability to it. As shown in Fig. 4, this means that Ensemble Set can fail improve once it reaches some level of human help. Second, it requires 20x inference time compared to other methods, Investigating how to lower the computational burden and combining ensemble-base prohabilities with CP can be a fruitful future direction.
Task success
0.8
0.8
0.7
lower
0.7
lower
0.6
0.6
2
0.2
0.4 0.6 0.8 Human help rate
Average prediction set size
KnowNo
Simple Set Binary
Ensemble Set No Help
* Prompt Set *
Figure 4: Comparison of task success rate average prediction set size (Left) and vs. human help rate (Right) in Simulation averaged over the three settings. 200 trials are run for each method. c is varied from 0.25 to 0.01 for KNOwNo No, and from 0.6 to 0.01 for Simple Set and Ensemble Set. Binary and No Help are not shown the left since prediction sets are not provided.
4.2 Hardware: Multi-Step Tabletop Rearrangement
In this example, a real UR5 robot arm is asked to sort a variety of toy food items on a table (Fig. 5). In each scenario, three items are placed on the table initially, and the task is to sort them based on human preferences; we simulate a human with strong preferences for healthy food like eggs and fruits, and dislike for less healthy food like donuts and Skittles candies. To introduce ambiguities, the context for the LLM reveals only subset of the preferences. Here we consider a multi-step setting with possibly multiple acceptable options per step the LLM plans the new step conditioned on the previous action taken.
corn, celery, pear. peanut butter, sunny-dide-up egg don'' like pretzel, cracker, mustard, peach, MSM.
0.18 put chessee in green plate
0.54 put blue plate
you things and dielkk green plates?
0.07 - put donut green plate
0.03 an option not listed here
pear.
quantile: 0.23
0.54 put put poar blue platn
Step
the frrst stap put pear in the blu plate. Now. vill
Good, I like pear:
Pass mango,
0.25 put cheese in gr ewn plate
you cheese
0:10 donut in blke plate
Which
put green
No. don't lie dairy.
0.54 put chessse blue plate
0.10 an option not listed here
am at the side.
Step
CP quantlle:
0.23 Prodiction set: put choese green put cheese blue plate
Okey, will pick the mango with my right
Figure 5: (Lefi) Multi-step CP is applied in Hardware Tabletop Rearrangement. (Right) CP models ambiguity in possible human locations and triggers clarification from the human in Bimanual.
KNOWNO reduces step-wise and trial-wise inter- vention rate in multi-step setting. Since Section 4.1 has shown that Ensemble Set can be expensive (even more so in the multi-step setting) and Prompt Set and Binary can fail to achieve the user-specified success level, we focus on comparing KNOWNo with Simple Set for the remainder of the evaluation. Here we set the desired error level c= 0.25. Since Simple Set does not provide coverage, we first find <= 0.44 for Simple Set that achieves the same planing error rate as KNOWNo in simulation. Then we run 50 trials for both methods with real hardware. Table shows KNOwNo reduces the human help rate by 14% step-wise and 8% trial-wise, while also reducing the average prediction set size. Compared to Simple
Method 1 e Plan Suce Task Succ Set Size Help-Stop Holp-Trial
KNowNo 0.75
0.76
0.74
1.72
0.58
0.92
Simple Set
0.72
2.04
1.00
No Help
0.41
0.38
0
0
Table 1: Results for Hardware Multi-Step Tabletop Rearrangement, Plan success rate is fixed between KNOWNO and Simple Set.
7

---

Set that uses a much higher e, KNOWNo achieves trial-level task success rate precisely leveraging the Multi-Step Uncertainty Alignment from Sec. 3.3. We also find that if we set 0.2. for Simple Set, the planner is grossly over-conservative and requires a step-wise help rate of 87%.
Bimanual manipulation. We additionally present results for a bimanual object rearrangement setup where ambiguities arise from the choice of the arm (Fig. 5 right); results are deferred to Section A4.
4.3 Hardware: Mobile Manipulator in a Kitchen
Method
Model
Suce Suce Size Help
KNowNo PaLM-2L 0.85
0.87
0.76
2.22
0.67
In this example, each scenario involves a mobile manipulator in front a countertop and next to a set of pecyclin/ccomoo//aamngg
Simple Set PaLM-2L 0.76
0.87
0.75
2.38
0.81
Help PaI.M-21.
0.62
0.51
bins in a real office kitchen (Fig. 1). The tasks include pick- ing up some object on the counter, and possibly success rate is fixed between KNOWNO and Simple Set. putting it in the drawer, or disposing of it in one of the bins. For the distribution of possible scenarios, we introduce new types of ambiguities based on Winograd Schemas [18] (c.g. "There is the apple and bottled water on the counter...it is rotten. Can you dispose of it?"), and ones that potentially involve unsafe actions (e.g.. "place the bowl in the microwave."; ; there is a plastic bowl and a metal bowl, but only the plastic one safe for the microwave). This is a single-step setting with multiple acceptable options. In Table 2, we compare KNOWNO to Simple Set again by first setting e = 0.15 and finding 0.24 for Simple Set that achives the same plan success rate. The hardware experiment results again show that KNOwNo reduces the human help rate by 14% and also reduces the average prediction set size.
KNOWNO GPT:3.5 0.85
0.87
2.50
0.86
Table 2:
Results for Hardware Mobile Manipulation. Plan
Target success guarantee from KnowNo is robust to varying LLM choice. We also run KNOWNo with GPT-3.5 (fext-davinci-0033 from OpenAI (without hardware evaluation). However, we find that it exhibits sigmificant MCQA bias towards options D and E and against A and B, affecting the overall performance. Nonetheless, KnowNo still achieves 1 - target success rate, as the coverage guarantee from CP makes no assumption about the LLM confidences (e.g., calibrated or accurate) - KnowNo flexibly compensates for the degraded LLM performance by triggering more human intervention.
5 Related Work
Uncertainty quantification for LLMs. Motivated by LLMS' overconfidence and hallucination, there has been a growing body of work in quantifying and better calibrating uncertainty [19, 20, 21, 22, 23, 24, 25]. In contrast to typical calibration methods that associate uncertainty with point- valued outputs, CP-based methods for language modeling provide coverage guarantees for set-valued predictors [26, 27, 28, 29].
Conformal prediction in robotics. To the best of our knowledge, this work is the first to employ CP for languag-bassed planning. Prior work has utilized CP for fault detection, trajectory prediction, and planning in dynamic environments [30, 31, 32, 33]. We provide a novel multi-step extension to CP to guarantee correctness for the entire planning horizon by performing sequence-level calibration in settings where the robot's actions influence the distribution of future inputs.
fluman-robot dialogue and interaction. KNOWNo builds off previous work in robotics that addresses effective human-robot interaction through dialogue [34, 35, 36, 37, 38]. KNOwNo uses a relatively simple setup such that the human specifies potentially ambiguous instruction and clarifies it when necessary; such setup does not consider human providing information related to robot observation or possible human error.
6 Discussion
We propose KNOWNO, framework that applies conformal prediction (CP) to address the problem of uncertainty alignment for guarantees of task completion while minimizing human help. Experiments across a variety of simulated and hardware setups demonstrate that KNOWNO achieves user-specified task completion levels consistently while reducing human help by 10 10-24% compared to baseline approaches that lack formal assurances.
Summary:
instructed robots, which we formalize as providing statistical
Limitations and future work: The primary limitation of our work is that the task completion guarantee assumes environments (objects) are fully grounded the text input to the LLM,, and the actions proposed by the LLM planner can be executed successfully, In the future, we are looking to incorporate uncertainty of the perception module (e.g., rision-languaa model) and the lewelevel action policy (e.g., Ianguage-condttoon affordance prediction) into the CP calibration.
8

---

Acknowledgments
This work was partially supported by the NSF CAREER Award [#2044149] and the Office of Naval Research [N00014-23-1-45 1-2-444 We thank Chad Boodoo for helping set up the URS hardware experiments, and Jensen Gao, Nathaniel Simon, and David Snyder for their helpful feedback on the paper.
References
[1] M. Ahn, A. Brohan, Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as I can, not as say: Grounding language in robotic affordances. arXiv preprint aXXiv:204401669, 2022.
[2] W. Huang. F. Xia. T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, L. Mordatch, Y. Chebotar. Sermanet, T. Jackson, N. Brown, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning, 2022. URL
wwt//eeennrnn wwwrrrnnnddddddadaeeeee d-33325500ty..
[3] V. Vovk, A. Gammerman, and G. Shafer. Algorithumic Learning in a Random World, volume 29. Springer, 2005.
[4] A. N. Angelopnulos, S. Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trendst9 in Machine Learning, ((4))494-591, 2023.
[5] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florenee, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2009007553, 2022.
[6] K. Murray and D. Chiang. Correcting length bias in neural machine translation. arXiv preprint arXiv:1808. 10006. 2018.
[7] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alone et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. 2022.
[8] D. Hendrycks, C. Bums, S. Basart, A. Zou. M. Mazeika. D. Song. and J. Steinhardt. Measuring massive multitask language understanding, arXiv preprint arXiv:2009.03000 2020.
[9] V. Vovk. Conditional validity of inductive conformal predictors. In Asian Conference on Muchine Learning, pages 475-490, PMLR, 2012.
[10] M. Sadinle, J. Lei, and Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, (14652))223--34, 2019.
[11] Z. Jiang, J. Araki, H. Ding, and G. Neubig, How can know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 962-977, 2021.
[12] Y. Bai, S. Mci, H. Wang. Y. Zhou, and C. Xiong. Efficient and diffcrentiable conformal prediction with general function classes. arXiv preprint arXi::2002.1 1091, 2022.
[13] Google. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos. S. Shakeri, E. Taropa, P. Bailey. Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstem, G. Mishra, E. Moreira, M. Omemick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Xu, Y. Zhang, G. H. Abrego, Ahn, J. Austin, Barham, J. Botha, A. Chowdhery, C. Crepy. S. Dave, M. Dehghani, S. Dev, Devlin, M. Diaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur-Ari, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycheriah, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, Li, Li, Y. Li, Li. Lim, Lin, Z. Liu, Liu. M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado, J. Nham, E. Ni, A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope. S. Qiao, E. Reif, B. Richter, P. Riley, A. C. Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. So, D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang. 7. Wang, T. Wang, J. Wieting, Y. Wu, K. Xu, Y. Xu, I. Xue, Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. 7hou, D. Zhou, S. Petrov, and Y. Wu, PaL.M technical report, 2023.
J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Chemy,, C. A. Choquette-
[14] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv: 2021.
[15] C. Guo, G. Pleiss, Y. Sun, and K. Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.

---

[16] A. Angelopoulos, S. Bates, J. Malik, and M. 1. Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009. 14193, 2020.
[17] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning.
org. 2016-2022.
[18] H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
[19] K. Zhou, D. Jurafsky, and T. Hashimoto. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302.1343) 2023.
[20] Y. Xiao and W. Y. Wang. Quantifying uncertainties in natural language processing tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7322-7329, 2019.
[21] Y. Xiao, P. P. Liang, U. Bhatt, W. Neiswanger, R. Salakhutdinov, and L.-P. Morency. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.04774, 2022.
[22] I.. Kuhn, Y. Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXi::330.006664 2023.
[23] S. Kadavath, T. Conerly, Askell, Henighan, D. Drain, E. Perez, N. Schiefer, Z. Haffield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hemaandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan. Language models (mostly) know what they know. arXiv preprint 2rXiv:220..05221, 2022.
[24] Lin, Hilton, O. Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 14334, 2022.
[25] S. J. Mielke, A. Szlam, E. Dinan, and Y.-L. Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Trunsactions of the Association for Computational Linguistics, 10:857-872, 2022.
[26] T. Schuster, A. Fisch, T. Jaakkola, and R. Barzilay. Consistent accelerated inference via confident adaptive transformers. arXiv preprint arXiv:204.0088e 2021.
[27] Schuster. A. Fisch, Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:1746611772,, 2022.
[28] N. Dey, J. Ding, J. Ferrell, C. Kapper, M. Lovig, E. Planchon, and J. P. Williams. Conformal prediction for text infilling and part-of-speech prediction. The New England Journal of Stutistics in Data Science, 1(1):69-83 83, 2022. ISSN 2693-7166. doi:10.5138770 NES/SS2
[29] Giovannotti and A. Gammerman, Transformer-hssed conformal predictors for paraphrase detection. In Conformal and Probabilistic Prediction and Applications, pages 243-265. PMLR, 2021.
[30] R. Luo, S. Zhao, J. Kuck, B. Ivanovic, S. Savarese, E. Schmerling, and M. Pavone. Sample-efficient safety assurances using conformal prediction. In Algorithumic Foundations of Robotics XV, pages 149-169, Cham, 2023. Springer International Publishing. ISBN
[31] L. Lindemann, M. Cleaveland, G. Shim, and G. Pappas. Safe planning in dynamic environments using conformal prediction. arXiv preprint arXiv:2210. 102544 2022.
[32] L. Lindemann, X. Qin, J. Deshmukh, and G. J. Pappas. Conformal prediction for STL runtime verification. In Pruceedings of the ACM/EEE 14th International Conference on Cyber-Physical Systems (with CPS-loT Week 2023), pages 142-153, 2023.
[33] A. Dixit, L. Lindemann, S. Wei, M. Cleaveland, G. J. Pappas, and J. W. Burdick. Adaptive conformal prediction for motion planning among dynamic agents. arXiv preprint 12.27 2022.
[34] Tellex, R. Knepper. A. Li, D. Rus, and N. Roy. Asking for help using inverse semantics. 2014.
[35] K. Nguyen and H. Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiesit--encouraging imitation learning. arXiv preprint (tt::19990008777 2019.
[36] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and and---nd-dii -dialog navigation. In Conference on Robot Learning, pages 394 406. PMLR, 2020.
10

---

[37] A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan-Chen, S. Gella, R. Piramuthu, G. Tur, and D. Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 2017-2025, 2022.
[38] X. Gao, Q. Gao, R. Gong. K. Lin, G. Thattai, and G. Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. IEEE Robotics and Automation Letters, 7(4):10049- 2022.
[39] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, L. Misra, and N. Carion. MDETR-modulated detection for and-to-end multi- modal understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1780-1796 2021.
[40] M. Minderer, A. Gritsenko, Stone, M. Neumann, Weissenborn, A. Dosovitskiy, A. Mahendran, A. Amna, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection with vision transformers. arXiv preprint arXiv:205.0062 06230, 2022.
[41] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y Lo,
P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643, 2023.
[42] V. Agafonkin. Polylabel: a fast algorithm for finding the pole of inaccessibility of polygon, July 2016. URL
com womm//appox/po//////cc
[43] S. Singh, J.J. Sloune, and V. Sindhwani. Optimizing trajectories with closed-loop dynamic SQP. In 2022 Interational Conference on Robotics and Automation (ICRA), pages 5249-5254, IEEE, 2022.
[44] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog,
J. Hsu, et al. RT-1; Robotics transformer for real-world control at scale. arXiv preprint 22X00.221..06817, 2022.

---

A1 CP in Multi-Step Setting
Algorithm Multi-step L.LM planning with human help.
1: for time t+- to ''-1 do Observe imput sing
3:
Predict a set ((aie)
4:
if ((iiss) is singleton then
5:
execute corresponding action
6: else
7: ask for help, which always assumed to provide the correct label, or the humnn provide clarification only
8: end
9:
end for
Proof of Claim I: Suppose 7 C (TTea)) We have,
TE (Tees)
< min f(tlea yr 2 21-
(AI)
(rready 21-9,
[
(A2)
- of E EC' (xfte))
VTE
(A3)
y E (Sress)-
(A4)
Proof of Proposition 2: Since we can bound the probability that Vue & C(Tres), we can also bound the probability that Vust $ (TTea). From the conformalization procedure, we have the following dataset- conditional guarantee: with probability - over the sampling of the calibration set 7, we have
P Fees E C (Trea)
Beta - (6). [(N+1)
(A5)
Claim
TJess E (Thret))
Beta
(8).
(A6)
where è is chosen such that c= 1 Beta N +1 +1 (0). Hence, the following marginal guarantee also holds:
IP Thee E C (Frest)
1-
Claim
(Tuest E (T 21 E.
This result provides bound on the task completion rate if Tist is drawn using the distribution D. However, recall that the sequence E of augmented contexts as defined in Section 3.3 arises from having performed the correct actions previous steps; incorrect actions may result in distribution shift. order to obtain bound on the task completion rate, we consider three cases at any given timestep: (1) the prediction set is singleton and contains the correct label, (2) the prediction set is not singleton but does contain the correct label, and (3) the prediction set does not contain the true label. The robot performs the correct action in the first two cases (without help in (1) and with help (2)), while CP bounds the probability of case (3). Thus, the CP bound translates to a bound on the task success rate.
As seen in Eq. (3), we have from [10, Thm. 1]. that we achieve the smallest average set size among all possible sequence-level prediction schemes, C, if models the prediction uncertainty accurately,
min E CEC (E,
(C ]. subject to P(Te ((F) 21-
(A7)
A2 CP in Settings with Multiple Acceptable Options Per Step
Proposition 3 (Multi-label uncertainty alignment) Consider a setting where we use CP with coverage level 1 -e to construct the prediction set when there are multpple true labels and seek help whenever the set not singleton at each timestep. With probability 1-8 8 over the sampling of the calibration set, the task completion rate over new test scenarios drawn D is at least 1-c.
Proof: We have datuset of Z Y(..,, sampled i..dd from data distribution D for calibration
(we use the same notation D the single-label setting here), where Y == ( the set of true labels for single trial. For each label, use the same heuristic notion of confidence, f(x), [0.1].
12

---

We define an operator 3:X'''' +y where A' is the space of contexts and y is the space of labels:
(xxY))= (1)
(A8)
yEY
which takes the true label with the highest confidence value from the true label set.
If we consider applying 3 every point in the support of D, distribution DD' is induced. We also consider the induced dataset of samples S' = (x ) where y; y == (1). Then we can
perform the usual conformalization and obtain the guarantee that with
((iiea) := [ul] (3 114))
(A9)
the following marginal guarantee holds,
P(yeet maxx & (Tress) Sc,
(A10)
P(argmax (Tea) & ( Se, yeYa P(3( & ((Tis))) Sé,
(A11)
(A12)
and the following dataset-comditional guarantee holds when we choose è such that c= 1=1- Beta -1 where v= [[N++))ē
1-Bea
-+1-r,
P( P( Yoss) (Z(ea)) 2-1
(A13)
Hence, (Tess) contains the true label with the highest confidence with probability at least 1-c.
At test time, we sample (Frest Yws) from D that is i.idd with samples in S - for the guarantee to hold for ( Y fes)), we need to show 3 (3 kes) is a sample from D' that i.i.d. with samples S". This is true since functions of independent random variables are independent, and functions of identically distributed random variables are identically distributed if the functions are measurable.
A3
CP in Multi-Step Setting with Multiple Acceptable Options Per Step
Proposition 4 (Multi-step, multi-label uncertainty alignment) Consider multi-step setting where we use CP with coverage level 1-e to causally construct the prediction set when there may be multiple true labels any step and seek help whenever the set is not a singleton each timestep. With probability 1-8 over the sampling of the calibration set, the task completion rate over new test scenarios drawn from ' is least c-c.
Proof: For the multi-step setting, each trial now involves a sequence of contexts 7 and a set of sequences of true labels:
Y (T.........ubb
(A14)
where Un. :- T= Wwnn. T T- For example, Y can contain the sequence of "blue block, yellow block, green block", "green block, blue block, yellow block". ..., for the task of picking up three blocks. We collect a dataset of Z {TT,,, {)} of i.dd samples from the data distribution D.
Unlike the single-step setting, here cannot apply , to the set of true labels in each step since we reasoning over a set of sequences, and not a sequence of sets of true labels. Notably, the true label set at time step f depends upon the sequence of previously chosen true labels.
sets
Let (²(=²" "] denote the set of true labels at timestep i, conditioned upon the initial context . and partial sequence of past true labels 1 (y ......y"-) the following sequence:
extracted from Y. We then autoregressively define
Fo = argmaax arggmax((0))
(A15)
yEYv
3. 3
argmaxx
((x) 1=1...7--1
(A16)
For convenience, we denote (x) the element in (x, An intuitive interpretation is that, we can consider Y forming a tree of valid executions (all possible actions that can be taken by choosing each of true labels). Hence, at each time step i, (T,Y) prunes the tree to single branch by taking the true label with the highest heuristic value f(x"). This reduces the tree of all possible sequences of true
13

---

labels to a single branch of true labels with highest confidence. Given this single branch of true labels, we can now perform CP as shown in the multi-step setting in Section Al.
We apply 3T-1 to every point in the support of D, and new distribution D' is induced. We consider 5' = [( "), where y, == - a). Let V Yeu be the set of sequences of true labels for Trst- Suppose we get the marginal bound with _ as the labels:
(T ws) & C(Trea) SC,
(A17)
and dataset-condditooaal bound when we choose such that c=1 ---BBaa -Beta (6) where e= ((N+))
P P( (Zusa Y tea) & (ees (2)
(A18)
which states that at test time, given a context sequence Tusa we produce a prediction set of sequences; we consider a sequence consisting of the true label with the highest score at each step, the probability of this sequence covered by ((Trest) is lower bounded by 1 cc However, we need to be careful of following
3, at each step at test time, Conside the three cases:
(1) At given time-step. the prediction set (*(laa)) does not contain the true label, (x.)))
(2a) The prediction set a singleton and does contain the true label.
(2b) The prediction set is not singleton (but does contain the correct label).
We already bound the probability of (1) happening with the CP bound; (2a) is fine since the LLM will take the correct action; (2b) more challenging - in this case the robot asks the human for help, and we need to make sure the human "follows" the true label, by choosing the true label in the prediction set with the highest confidence by f. In practice, we present the labels ranked by f and ask the human to choose the true label with the highest rank.
Now let's derive the bound Eq. (A17) and Eq. (A18). Again we need to consider the causal construction issue. As seen in Section 3.3, we construct the prediction set ((Tss) non-causally using the score function 8, = (Z) (taking minimum over steps). For a test sequence Trest, we apply 8 to the true label set of sequences Y Yest to get Viest 3 (Frest, Y Ieu)). Now suppose Vest E (Tree) ). then we can show Fest EC (Tiee) with the same proof as the single-label setting, which gives us the bound.
Lastly need to show the sampled test sequence from D leads a sample from D' iidd. with S'. This is true with the same argument that functions of independent random variables are independent.
A4 Additional Experiment Setting: Hardware Bimanual Setup
In this example, a real bimanual setup with two Kuka IIWA 7 arms move objects the table, with one bin at each side (Fig. 5 right). The reachable workspace of each arm is limited so that one arm cannot reach the other end of the table the other bin. Thus, there can be ambiguities in the choice of the arm depending on the task; e.g., Fig. 5 shows the human asking the robot to pass over the mango, but not specifying where the human is. KNOWNO is able to capture such ambiguities and triggers clarification. We design a scenario distribution (single-step with single acceptable options) with all instructions being ambiguous (thus requiring high human intervention rate): with 0.15, the robot achieves 84% plan success with 92% help. With 10 real trials, the robot succeeds 9 times while triggering help for 9 times. Details of the scenario distribution are shown in Section A6.
A5
LLM Prompt Setup
Next we detail the LLM prompt setup for MCQA applied in KNOWNO. We will use the Mobile Manipulation setting from Section 4.3 as the example.
Multiple choice generation. Given a scenario, we first prompt the LLM to generate four options for possible actions. We apply few-shot prompting as shown in Fig. A1 below with zero temperature. In this scenario, there a Coke, a bottled tea, and a Pepsi on the counter, and the task is to put the Coke in the top drawer but the choice of drawer is under-specified Put the Coke in the drawer please.").
14

---

Ne: You are rebot operating in an office kitchen, You are in front of counter with two closed drawers, top one and niddle one, There is also a landfill bin, a recycling bin, and a compost bin.
No: On the counter, there is orange soda, Pepst, and an apple. draner.
Ne: Put that drink
A) open
put orange open the niddle drawer and put the Pepat 11
top
in
C) ogen the niddle drawer and put the orenge sods in it
D) open the top drawar and put the Pepsi in 1t
No: On the counter, there is energy her, banana, and microwave. the nicrowave.
Ne: Put the snack
A) pick up energy
put it next B) pick up banana and put next to the anergy
microwave
C) pick up the benana and put it next to the nicrowave
D) pick up the energy bar and put it next to the banane
No: On the counter, there is Cake, Sprite, and sponge.
Ne: Can dispose of the can? It should have expired.
A) pick up sponge
put
landfill recycling
B) pick up
and put
C) pick up the sprita and put it in the recyeling bin
D) pick up the Coke and put it in the landfill bin
No: On the counter, there is bottled water, bag of jalapano chips, and bag of rica chips. would like bag of chips.
A) pick up bottled water
pick up jalapano chips
C) pick up the kettle chips
D) pick up the rice chipa
(The correct option is either 8 or D. since either jalapeno chips or rice ships are fine.)
the counter, there Coke, bottled unsweetened and Pepsi.
Coke the drawer please.
You:
Figure A1: Prompt used for multiple choice generation in the Mobile Manipulation setting.
After the LLM generates four options, we append an additional option 'an option not listed here' to the four generated ones and then randomize the order to further prevent bias. We then use a zero-shot prompt Fig. A2 for querying next-token probabilities ('A', 'B', 'C', D', 'E').
You robot operating top one niddle one. There also landfill bin, recycling bin, compost bin.
office kitchan. in front of with closed drawers,
Ne: On the counter, there is a Coke, bottled unsweetened tes, and a Pepsi.
No: Put the toke in the drawer please.
You:
A) pick the coke
6) pick
and and
the draner botton drawer
C)
D)
option
here
E) pick up pepsi
Ne: Which option is correct? Answer with a single letter.
You:
Figure A2: Prompt used for next-token prediction with generated multiple choices in the Mobile Manipulation setting.
A6 Additional Experiment Details
Environments. In addition to Fig. I and Fig. 5, here Fig. A3 shows the office kitchen environment with the set of drawers and bins used in the Mobile Manipulator experiments (left), and the bimanual setup with the set objects used on the mat (right). There is another set of drawers used in the mobile manipulation experiments underneath a much bigger countertop not shown here.
Scenario Distribution and Calibration Dataset. Next, we provide more details on the parameterization of the scenario distribution in each experiment setting, in particular, the possible ambiguities with the instruction and goal. With the distributions set up, the calibration dataset are then generated by randomly sampling 400 i..d scenarios from them:
Simulated setting:
Environment: there are always three blocks and bowls of color red, yellow, and green with random locations on the table.
Goal: we use the following template: (put, place, move) (a, one, a single of, two, a pair of, three, all, red, yellow, green} (block(s), bowl(s)) (on, to the left of, to the right of, to the front of, at
15

---

Figure A3: (Left)) Office kitchen environment with drawers and bins for the Mobile Manipulation setting. (Right) Bimanual setup with the set of objects used in the experiments.
the back of} the fred, green, yellow) {block(s), bowl(s)). The scenario distribution uniform over these possibilities.
Instruction: for the language instructions, we modify the goal (from the template) according to the ambiguity type. The scenario distribution uniform over the listed ambiguous cases in each ambiguity type.
Attribute ambiguities: refer to the block one of "cube", "cuboid", "box", "square object", to the bowl as one of "container", "round object". "receptacle", or to either block or bowl as one of "object", "item", "thing" ('move the blue object in yellow bowl"); refer to "blue" one of "cyan"" "navy", to "green" as one of "greenish", "grass-colored", and to "yellow" as "orange" or "gold". This setting is the least ambiguous one among the three ambiguity types.
Numeric ambiguities: refer to either two or three numerically with one of few", "a couple of", "some", "a handful of" ("put some blocks in the green bowl").
Spatial ambiguities: refer to any of the four possible directions with "near", "close to", "beside", "next to",, refer to either left to right with "lateral to",, and refer to either front or behind with "along the line of sight". This setting is the most ambiguous one among the three ambiguity types.
Hardware Tabletop Rearrangement setting:
Environment: there are three items to be sorted placed randomly on the table, and there is blue plate and green plate. 28 toy items are split (Fig. A7) into two categories of human liking them or disliking them: the things the human likes include corn, avocado, celery, carrot, tomato, lettuce, apple, orange, pear, lemon, peanut butter, aunn--side-up egg, egg, and pea; the human dislikes pretzel, cracker, waffle, mustard, ketchup, pizza, meat patty, cheese, chicken drumstick, peach, mango M&M, Skittles, and donut. The three items the table uniformly sampled ones from the two categories (1 from likes and 2 from dislikes, or from likes and 1 from dislikes).
Goal: place the items that human likes in the blue plate, and ones that human dislikes in the green plate.
Instruction: to make the goal ambiguous, only subset of human preferences are revealed the prompt:"1 like: com, celery, pear, peanut butter, sunny-side-up egg. don't like: pretzel, cracker, mustard, peach, M&M" And then the instruction includes "Can you sort things like and dislike in blue and green plates?"
Hardware Mobile Manipulator setting:
Environment: the full list of possible objects include: bottled water, bottled tea, orange soda, RedBull, Coke, Pepsi, Sprite, rice chips, jalapeno chips, kettle chips, multigrain chips, apple, orange, energy bar, clean sponge, dirty sponge, metal bowl, plastic bowl. Depending on the ambiguity listed below, there is three objects placed on the top of the counter (including randomly sampled distractors from the list). There is also a set of landfill, compost, and recycling bins, a microwave, and a portable stove.
Instruction: for convenience, we introduce the possible instructions first different ambigu- ous scenarios; they each correspond to possible goals. Please refer to
help. github. i httpprpppp///mbblllclpp// hetiipptt//mobllleeeeee mewbbieetsssss.... tasks. txt for the full list. The possible instructions are uni- form distribution over different types: (1) single-label, e.g., 'Bring me a Coke' (unambiguous); (2) creative-single-lahel, e.g., want a healthy fruit to munch on." which means the apple (un- ambiguous); (3) multi-label, e.g., 'Biing me cola.' and either Coke or Pepsi is acceptable; (4)
16

---

creative-multi-kaae multi-label, e.g. 'Bring me something with a kick.' and either RedBull or jalapeno chips are acceptable; (5) aaaiigul,,,,,, e.g. 'Put the Coke in the drawer' or Put the Coke near the fruit' which under-specifies the drawer or fruit; (6) unsafe, e.g. 'Can you dispose of the bottle drink? It should have expired.' or 'Place the bowl on the stove, please.; (7) Winograd, e-g. 'There is sponge and bag of rice chips...I don't t want to use it for cleaning any more. Can you please dispose of it?" We use the GPT-4 model for generating the creative tasks.
Goal: the corresponding goal for the ambiguous instructions above. For example, the instruction is "Put the Coke in the drawer", and the goal is uniform over the two possibilities: put the Coke in the top drawer, and put the Coke in the bottom drawer.
Bimanual setting:
the full list of possible objects include: Coke can, Sprite can, green apple, banana, mango, red block, green block, yellow block, pink plushie, yellow plushie, purple plushie. Depend- ing on the ambiguity listed below, there is three objects placed on the table (including randomly sampled distractors from the list). There is possibly a human standing the left or right side of the table.
Enviromment:
Instruction: Please refer https: haap::://r/otteeeee help. txt for the full list of scenarios sampled from the distribution. The possible instructions are a uniform distribution over different types: (1) 'Pick up the {object) and pass it to me. I am next to the bin.' (2) 'Pick up the (object) with the left arm.' (3) 'Put the (object) in the bin closer to it.' (4) 'Pick up the {object) with the arm closer to it.' (5) 'Pick up the {object].' (6) 'Pick up the
hell..gii gi thub - http//ropts//iimiie /bt///aaaaaaaaaa//aa//a///..aaaa tasks.
{object] at the handle.' (7) 'Move the {object} to the front of the table.' (8) 'Move the {object] on the sticky rubber mat to the front of the table.'
Goal: the corresponding goal for the ambiguous instructions above. For example, the instruction "Pick up the Coke can and pass it me. I am next to the bin.", and the goal is pick up the Coke can with the left arm and pass it to human, if human is at the left side, and with the right arm human is at the right side.
Dafine possthle anvironrent, goal, and instruction possible.envy I'three blocks and three bowls") possible. gwals. arath_insttuuttinn puput three blocks the green bowl': l'put three blocks in the green bowl', 'put blocks the green green boxl -l. "put two blocks in the blue bowl": I' l'put two blocks in the blue bowl', "put a pair of blocks in the blue bowl" -l,
howl", "put three cubes
od key: goal; value: 1ist of possible inatructions
Sarple scenarios by uniformly sampling fron possible envarammenss, goals, and instructions wwn.ceelibration.da
1888
colibration dataset
range(mun_calltn [nge:/mun_ssttbratonndddt))
env randon, wwwpccceeppsssble.eeeg.
goal randon.chesce(lsstt paser.cchhsofssstppsssbb. goals wath instructions. k))
inatruction randos, (possccopossbblepppp insttrtttnntto [ggaal) calibration desibrationddttsee.... appendd (env, geel, instruction))
Figure A4: Sample algorithm for defining the scenario distribution and generating the calibration dataset in the Simulation setting.
Next we provide more details on some of the baselines.
Baselines Ensemble Set. Our ensemble-batd based method is weaker method than the traditional model- based ensemble where multiple copies of neural network are trained and inferred with; however, this is infeasible with the LLM use. In work, we randomize over the few-shot examples in the prompt as the ensemble. We select a pool of 20 possible MCQA examples (see examples in Fig. A1), and then randomly sample certain amount from it for each inference. Note that in this case, Ensemble Set actually has advantage over KNOW No and Simple Set that, for the same data, it has seen many more examples than the fixed ones in the prompt used in KNOWNO and Simple Set. We only apply ensemble for next-token prediction; the same set of multiple choices generated is used.
Baselines - Prompt Set. First, multiple choices are generated in the same way as KNOwNo. Then LLM is prompted to generate the prediction set, with few-shot examples in the prompt showing the possible labels (Fig. A5). For example, "We: Which options are possibly correct? You: A, C, D.".
Baselines- Simple Set. Instead of generating multiple choices, the LLM is first prompted to give the most likely action (e.g.. "We: Put the Coke can in the drawer. You: I will" shown in Fig. A8). Then we attach the generated response to the same prompt, and ask L.LM to label "Certain/Uncertain: given few-shot examples (Fig, A6).
17

---

table
these objects: yellow bowl, bowl, blue block, green blook, green bowl,
yellow block.
ll: Now, Put the blue object at the right side of the grans-colored square abject.
You: These are sone options:
A] option not listed here
put blue block to the right green
C) put blue block to the left green bowl
D) put blue bowl to the right of green block
E) put blue black to the left of green block
He: Mhich options are possibly correct?
You:
Ne: On the table there are these abjects: blue bowl, green block, yellow howl, yellow block, blue black
green bowl.
Ne: Now, Put the
round objest at the left side of the greenish block.
You:
sone options:
A) put green bowl to the left yellow block
B) an option nat 1isted hare
() put green bowl to the left of green block
D) put green bowl to the left of blue bowl
F) put grean
left blue block
Whichh options are possibly correct?
You: C
Ne: the table there these objects: yellow block, yellow bowl, blue block, green block, blue bowl
bowl.
Now, put the cyan bowl behind the gold object.
You: These are sone options:
A) put blue bewl. behind yellow black
B) put blue bowl behind
block
put blue howl bahind yallow bool
optaon not listed here
E) put blue howl behind blue block
We: Which options are possibly correct?
You:
Figure A5: Prompt with few-shot examples (two shown) for LLM generating the prediction Prompt Set baseline.
Ne: On the table there are these objects: blue bowl, green block, yellow bowl, yellow block, blue block, green bowl.
le: New, Put the green round object at the left side of the greenish block.
You: I will put incertain
bowl to the left of yellow block.
cortasn/Uneartann:
Ne: On the table there are these objects: blue block, yellow bowl, green block, green bowl, blue bowl, yellow black.
Nl: New, Put the yellow bax in front of the eyan receptacle.
You: will put yellow block the front blue block tartain/Uncartann. Cartain
Ne: On the table there are these objects: blue bool, blue block, green bawl, yellow bowl, green block, yellow block.
No:
Put the yellow box at the right side of the navy recaptacle.
You: will yellow black to the right blue blook. Certain/Uncertaal
Figure A6: Prompt with few-shot examples (two shown) for L.M expressing binary uncertainty in Binary baseline.
A7 Additional Implementation Details
While the focus of KNOWNo No is mainly on providing uncertainty alignment for the LLM-based planner, below we provide details of the perception and action modules applied in all examples.
Perception. For all settings except for the Mobile Manipulator, we either MDETR [39] (URS tabletop setting) or Owl-ViT' [40] (Simulated and Bimanual settings) open-vocabulary object detector for recognizing the objects in the environment and obtaining the object locations for low-level action. In Simulated and Bimanual settings, the variations of the object types are limited, and with general prompting. the objects are detected without issue. In the URS tabletop setting, since we are use a wide variety of toy items (Fig, A7 right), the detector has issues often differentiating objects like peanut butter and meat patty that are both darker colors. We modify the scenario distributions to avoid using such items together in one scenario. In addition, we apply the Segment Anything model [41] to extract the object segmentation masks (shown overlaid Fig. A7 left), and then use the polylabel algorithm [42] find the most distant internal point of the mask as the suction point (shown as red dots).
Low-level action. In Simulated setting and UR5 tabletop setting, simple pick-and-place actions are executed based on object locations and solving the inverse kinematics. Bimanual setting, the reachability of the Kuka arm is limited, and the pick-and-place action trajectories are solved using Sequential Quadtratic Programming (SQP) instead [43]. In the Mobile Manipulator setting, for most of the tasks that involve simple pick-and-ple andce and opening the drawers, the action is from end-to- and-to-end policy from the RT-1
18

---

paN 162
Figure A7: (Left) MDTER [39] object detection with Segment Anything [41] and most distant internal point (red dots) for URS tabletop setting. (Right) The total 28 toy items used for the experiments.
policy (please refer to [44] for details), which takes in the raw observation. For some of the hard tasks such as putting the plastic bowl the microwave and putting the metal bowl on the bowl, object locations are assumed known and we scripted action policies.
Human feedback. In KNOWNO, once human help is triggered, human is presented with the prediction set to choose the correct action from (if there is one). For example, the prediction set could include 'A) put peanut butter in blue plate' and 'C) put peanut butter in green plate' in Hardware Tabletop Rearrangement In practice, we can convert the prediction set to a question in more natural language, c.g., "Do you like peanut butter or not?" using simple heuristics. In Mobile Manipulation and Bimanual, prompt the LLM to generate the question based on the prediction set.
A8 Additional Discussions
Sentence-level score leads worse performance. Section we hypothesize that the distribution of probabilities of LLM outputs p(y) is highly sensitive to the output length. Here we explore the effect of using sentence output and the perplexity score for CP in Simulation. We still apply multiple choice generation first to obtain the possible options from LLM, and then query LLM scoring, for example, the probability of "put the blue block in the green bowl" with the prompt ending with will" (Fig. A8). We test two different method: CP with RawProb, which uses the raw product of conditional probabilities p(y) = -T==pp(; O1.1...0--11. and CP with Perplexity, which applies length normalization and commonly used language modeling, perplexiy(y) exp(- Σ log p(a; 01.-..-(-11]- Table A1 shows that for all three settings, using either sentence-level score leads worse performance, and performance degradation correlates with variance of the multiple choice lengths. We also notice that there is no significant difference in performance between CP with RawProb and CP with Perplexity; this could indicate that beyond eliminating length bias, applying MCQA setup in KNOWNO affords additional benefit of soliciting more accurate confidence from the LLM.
Setting Multiple Choice Length Variance
Method
Set Size Help
KNowNo
1.18
0.18
Attribute
1.52
CP w/ RawProh
1.33
0.32
CP w/ Perplexity
1.35
0.36
KNowNo
2.23
0.69
Spatial
2.81
CP w/ RawProb
2.50
0.82
CP w/ Perplexity
2.67
0.88
KNOwNo
2.17
0.79
Numeric
8.51
CP w/ RawProb
4.06
1.00
CP Perplexity
4.06
1.00
Table A1: Comparison of KNOwNo with CP with sentence output and perplexity score in the three ambiguity settings in Simulation. 0.15.. See Fig. A9 for samples of multiple choices generated with varying lengths.
Potentially stronger baselines with model fine-tuning. In Section 4 we introduce the two prompt- baselines Prompt Set and Binary, and demonstrate them being (1) inflexible (not allowing controlling the target success rate) and (2) do not properly model the uncertainty. We note that these two baselines can be potentially strengthened by fine-tuning the LLM to better predict the binary uncertainty the
19

---

No: You are rabot, and are asked to nove objects to precise locations on the table. Our instructions anbiguous.
there are these objects: green block, yellow bowl, yellow block, boul, blue block, grean howl.
the
le: Now, put two blocks at the blueish bowl.
You: I will put green block and yellow block on blue bowl.
Ne: On the table there are these objects: yellow bowl, blue howl, green block, yellow block, blue block, bowl.
Now, put yellow bool
couple of receptacles the right side of the blue block. block.
You:
bowl the right
We: On the table there are these objects: green block, yellow bool, yellow block, blue bowl, blue block,
green bowl.
il: Now. put one bowl st the front of anber-colored black.
You: will put green howl at the front of yallow blook.
Ne: the table there these objects:
block, blue block, bowl, yellow bowl, blue boxl,
yellow black.
Now, put cuboids at the right side the yellow round object
You: I will
Figure A8: Prompt used when evaluating the sentence-level scores in Simulation. We first generate the multiple choices in the same way as KNOwNo, and then evaluate the score of each multiple choice with this prompt. This prompt is also used generate the option in the Binary baseline.
uncertainty set, if the true labels can be properly defined. In fact, some recent work [24, 23] have explored model fine-tuning and exhibiting the effectiveness of Binary for uncertainty calibration. We also explored fine-tuning the GPT3 model (davincí) from OpenAI, which is the most powerful one from OpenAl available for fine-tume tueintunng....o However, we find the model performing at very low accuracy with MCQA, and fine-tuning the model always results overfitting to the dataset, even with thousands of data and varying hyperparameters (including ones from [24]). We suspect that our scenarios exhibit high complexity and variance, and it is non-trivial to fine-tune the model well with dataset. Nonetheless, we do hope to have future work looking into better training the model for proper uncertainty, and then applying CP on top of it achieve set-based calibration.
Attribute anbiguity
table are these objects: green block, blue bowl, yellow block, green bowl, yellow bowl, blue block.
ll: Now, put the yellow square object at the right side of the arange container.
You:
A) put yellow block the right of
boul B) put yollow block the right yellow bowl
C) put yellow block the right blue bowl
D) an option not listed here
E) put yellow block to the right of blue block
Nuneric anbiguity
table green bowl.
are these objecte: yellow bowl, blue bowl, green block, yellow block,
lle:
Now, put a handful of receptacles at the right side of the blue block.
You:
A) option not listed here
put yellow
put green block yellow blook right of block
bowl
right blue blook
D) put yellow bowl and blue bowl and green bowl to the right of blue block
E) put green block and yellow blook and blue block to the right of blue block
Spatiel aubiguity
table blue block.
are these objects: green bowl, yellow bowl, bowl, yellow blook, green
lla Now, put the yellow block along the 1ine of sight of the blue square object.
You:
A) option not listed here
put yellow block
blue block
put yellow block the left
block
D) put yellow block in front of blue block
E) put yellow block to the right of blue block
Figure A9: Sample multiple choices generated with Attribute, Numeric, and Spatial ambiguities Simulation.
20

---

Attribute
Numeric
Spatial
Task success rate
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.6
0.5
0.5
0.5
size
1.0
Task Task success rate
0.9
0.9
0.8
0.8
0.7
0.6
0.6
0.6
0.5
0.5
0.8
Human help rata
KnowNo
Simple Set
Ensemble Set
Prompt Set
Binary
No Help
Figure A10: Comparison of task success rate vs average prediction set size (Top row) and vs. human help rate (Bottom row) in Simulation for each of the three ambiguity settings: Attribute, Numeric, and Spatial (Columns), as varies from 0.25 to 0.01 for KNOWNO, and from 0.6 to 0.01 for Simple Set and Ensemble Set. Binary and No Help top row since prediction
shown
not involved. Prompt does not show up the human help rate plot for the Spatial setting as it only achieves 35% task success. As the setting gets more ambiguous (least in Attribute and most in Spatial), the performance difference between KNowNo and Simple Set grows.
21

---

the counter,
RecBull,
unsveetened tea, bag multigrein chips.
drank already. Can you dispose
You:
A) an option not listed here
B) pick up the redbull and put it in the landfill bin
C) pick the bottled unswee tened Les and put it in the landfill bin
D) pack the bottled unsrestened tea and put it the recycling bin
E) pick the redbull and
the recycling bin knowlo:: ['D' Prediction set with Simple Sat: I'0',
label: ['b']: Pradiction
the counter, there is bottled tea, RedBull, and Pepsi.
Ne: Bring ne sonething to drink.
You:
A) an option not listed here
B) pick the redbull
6) pick the peps1
D) pick the bottled Lea
bottled tea and
the recycling bin
label: ['B', 'C', 'D']; Prediction with Knowlo: ['D']; Prediction Simple (MA",
Ne: on the counter, there is Sprite, a bag of kettle chips, and a toke.
Ne: Put the soda in the drawer.
You:
A) open the top drawer and put the cake in 1t
B) an option not listed here
c) the middle drawer and the coke in it
open the middle drawer and
E) open
put
True label: l: Predaction
KnowNo: I'A'l; Prediction set wath Simple Set: I'A', 'E"1
Ne: On the counter, there is an apple, a orange, and an energy bar.
Nl: Bring me fruit.
You:
A) option not listed here orange
pick energy bar
pick
E) pick the apple and put it in the landfall
True label: ['B', I;',; Prediction set with KnowNo: ('B"]; Prediction set with Simple Set: UWA,, 'B"]
We: On the counter, there is an apple, bag of nultigrain chips, and bag of rice chips.
Nl: want healthy snack.
You:
A) pick the rice chips
nultigrain chips apple
pick
D) pick apple and put it the conpost bin
E) an option not listed here
True label; ['B", 'C'l, Prediction set with Knowtoo: ['c' I; Prediction set with Simple Bet: ['A', 'o"]
Ne: On the counter, there is an orange soda, Sprite, and Pepsi.
Il: Can you away the soda? It should have expired.
You:
pick
and put it the recyoling bin listed
an option
() pick the sprite and put it the recycling bin the cospost bin
D) pick up the orange soda and put
E) pick up the orange soda and put it in the recycling bin
True label: ['E']; Pradiction set with Knowlo: ['E']; Prodsction set with Sirple Set: ['0', 'E']
the counter, there chips.
jalapeno chips, of nultigrain chips, and of rice chips.
Bring
pick
nultiorain chips pick the riee chips
() pick up the jalapeno chips
D) an option not listed here
E) pick up the jalapeno chips and put it in the recycling bän
True label: ["A". 'B". 'c'l: Prediction set with Knowlo: I'X'l: Prediction set with Simple Set: ['X", 'E')
Figure All:
Randomly sampled scenarios from Mobile Manipulation, where KNOWNo generates singleton prediction sets that contains one of the true labels and avoids human intervention, while Simple Set generates non- singleton prediction sets and asks for human help. In Table 2 we show that KNOWNo reduces the human intervention rate by 14%. We also point the third example where the instruction is 'Put the soda in the drawer', which is ambiguous the choice of top and middle drawer. In this scenario, it happens that the human means the top drawer, and KNowNo generates the prediction set that only include the option 'open the top drawer and put the coke This example exhibits the inherent bias in LLM (e.g., hias towards top drawer over middle drawer).
22


Reviews:
[{"review_id": "jCivTRvUKC", "reviewer": "Reviewer_fuC2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper looks at enabling robots to compare possible next steps in a task plan and use their calibrated uncertainty to indicate when to ask a question -- for example, to clarify some ambiguity in the scene. This is an important ability for our agents to have -- both for their own success and for alignment to human preferences.", "summary_of_recommendation": "I think this is a valuable direction for the community to be moving, and I think the proposals are reasonable for the scope of a paper -- but a few questions are left open and the work does not articulate how its claims/approaches are related to or differ from what is already in the literature.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "This brings me to my questions:\n\n1. How would a more diverse captioning (or sampling - e.g. https://arxiv.org/abs/1904.09751) change the results for the Simple/EnsembleSet and would that close the gap with the KnowNo approach?\n2. It is unclear to me how significant the differences in Table 1 and Figure 4 are?  How many examples separate these approaches?\n3. [nice to have] How is performance impacted by the scene/task complexity? Is part of the reason the approach works that it's easy to get good coverage over the space of next actions/objects?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The basic idea proposed is intuitive and appears to generally benefit system performance (aka successful task completion).\n\nIn addition to the questions below, my main concern with this paper is not being properly situated in the literature.  There is a long history of people working on robot dialogue both within simulation and on physical hardware, for navigation and manipulation.  I don't feel it's appropriate for me to do a full literature review here but CVDN, TEACh, Dialfred are recent in simulation, RMM, Jarvis are recent models, RobotSlang from CoRL 2020, and SigDial has run entire special tracks on robot dialogue etc, but none of this literature is cited here.  I think the authors can claim there are important differences about how their proposal is tied to calibration on LLMs, but there's also a large literature on language generation evaluations that's not cited here -- I will note there is more awareness of the LM literature in the related work, but not a direct discussion of how what's proposed here and the claims connect to standard practice, whether it's BARTScore, T5Score, etc which have lots to say about the issue of length penalties and so forth (both within semantic similarity monolingually and machine translation).  There are just a lot of places where the paper reads like the authors are the first people to look at these issues. The last literature missing is that of diverse generations, specifically diverse captioning, which a cursory read through CV proceedings will dig up a large literature -- likely very beneficial to the Ensemble Set definition.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "bPFSl0SY_Z", "reviewer": "Reviewer_mihK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents the framework KnowNo to provide statistical guarantees on task completion of a robot instructed by a large language model (LLM).\nTo this end, the paper incorporates the concept of conformal predictions into an LLM-based planning framework.\nWhenever the created set of predicted plans (with given task success level) contains more than one option, a human can be queried to resolve ambiguity. \nThe approach has been evaluated on multiple tasks in simulation and on a real robot. ", "summary_of_recommendation": "\nThe paper presents an interesting addition to LLM-based planning for robots by leveraging the powerful concept of conformal predictions.\nThe motivation is clear, and the authors provide a sophisticated framework with a great potential.\nYet, the approach is not well described and discussed.\nDetails that are crucial for the approach's robustness, such as how the calibration datasets are created, are not described in enough detail, and discussed.\nThe experimental validation, although on multiple tasks, is too simplistic to validate the advantages and claims of the paper.\nThe approach needs to be validated across scenarios with more objects and ambiguity, or on randomized tasks.\nLastly, the limitations section needs a major revision to properly discuss the approach's assumptions, how to set it up, when it works and when doesn't.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* What happens in situations where there is only one option? How does the multiple-choice answers work here?\n* What is the conclusion from the conducted experiments with various LLMs?\n* How are the calibration datasets generated? What assumptions are made and how can one determine the true labels?\n* How realistic is the assumption of having a \"unique correct candidate plan\"?\n* How does human help perform across all experiments?\n* What are the limitations of the approach?\n* Which assumptions does the approach make?\n* How does the approach perform on less simplistic scenarios and tasks?\n* How are preferences covered in the approach and in the calibration dataset?\n* How does the dataset-conditional guarantee work?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the challenging problem of dealing with \"confidently hallucinated outputs\" that occurs when leveraging LLMs to create robot plans.\nThe framework makes use of the recent advantages of applying the concept of conformal predictions to LLM outputs. \nThe idea is straightforward and clearly motivated. \nThe paper is generally well written, and readers can follow the proposed framework well.\nThe figures explaining the framework are helpful to understand various aspects of KnowNo.\nIt’s great to see that the framework has also been applied to real hardware.\nThe submitted videos nicely visualize and summarize the conducted experiments. \n\n\nAlthough the paper proposes an interesting framework using conformal prediction, the approach's technical description, experimental validation, and discussion open more questions than providing answers.\nConformal prediction is a powerful tool, but its coverage depends on the right calibration set.\nAt the moment, the paper does not describe how to create such calibration datasets.\nEspecially for language planning, creating calibration datasets is a tedious and time-consuming task. \nFor sequence plans, this process is even more tedious. \nHow can designers make sure that they created a good set?\nMoreover, the ambiguity of language makes it difficult to find the \"true\" label. \nThe assumption that there is a \"unique correct candidate plan\" is unrealistic for many language-based planning tasks and not applicable when one wants to use language-based planning.\nThis crucial part of the approach requires much more attention in the paper. \nAs indicated in the paper, coverage would require new fresh calibration datasets. \nTo alleviate this, the paper leverages a dataset-conditional guarantee (see Eq. 2) but does not go into detail about it.\nThis part requires more explanation and clarifications why it hold for the considered LLM-based planning application. \n\nThe paper nicely motivates the task success coverage by leveraging human inputs whenever there is ambiguity. \nThis idea is present in the introduction and in a short paragraph later.\nYet, the approach and the experiments itself do not really demonstrate and analyze how well the human help works.\nFor simple tasks (like the placing of plastic or metal bowl), humans might be able to help.\nBut in other instances, such as when considering preferences, the human help might result in problems later during sequenced plans.\nFor instance, due to a mismatch of what the human thinks the robot can do.\nIt would be great if the paper analyzes how well the human help concept works and across a randomized set of tasks.\nAlso, the concept of preferences is introduced throughout the paper but not really the focus of the original idea. \nIt disturbs the reading flow and is also not well evaluated.\nAnd, as described in the paper's discussion, preferences are not well covered yet.\nIf the paper wants to include preferences, it should also be validated against, e.g., preference-learning baselines.\nOtherwise, the paper can also remove the \"focus\" on humans and remove those paragraphs.\nThis would also save space to describe more important technical details.\nTo this end, the paper sometimes reads as sections have been written with different styles and levels of side notes.\nIn my opinion, many side notes (e.g., \"multiple acceptable options\") are thrown into the context without proper explanation and motivation.\nTo simplify the paper's flow, it would be better for readers if the paper focuses on the core idea.\nThe gained space can also help again to improve important details.\nMoreover, LLM-based planning will (in most situations) make use of sequenced planning (time series).\nSo why not directly describe this setting and not spend a lot of time on the single planning case and reiterating and abusing notations to introduce the sequenced version?\n\n\nThe experiments have been carried on multiple platforms and tasks, and mostly look good.\nYet, to my understanding, the experimental validation is often handpicked, and the considered scenarios are overly simple.\nFor instance, Exp. 1 considers scenarios with just one alternative (choose between two clear objects) and Exp. 2 focuses on preferences placing items in one of two possible goal locations with single planning steps instead of sequences. \nAre those the difficult hallucinated plan options that the introduction mentions?\nI wonder if these are the realistic problem cases where the approach is needed. \nThe experiments would better show the approach's benefits in situations where the robot has much more options to choose from or in situations where the LLM hallucinates actions and plan sequences. \nAn example could be to task the robot to set the dinner table or to clean up the office space. \nIn the experiment section, there is also no clear description of details, such as how the calibration datasets have been created or which scoring functions have been used.\nThis makes it difficult to reproduce the results of the paper.\nThe description and discussion of the results is short (probably giving the space constraints), and it is often difficult to follow the claims.\nThe provided numbers or briefly mention but not discussed well to draw proper conclusions, e.g., in Fig. 3: how many examples have been used?  and in Fig. 4: where is the binary label in the left figure or why are there continuous graphs for a discrete prediction set size?\nThe analysis of using other LLMs draws no conclusion.\nThe considered baselines are not well described, and it is not clear how they are exactly implemented and why they have been chosen.\nFor instance, the ensemble set seems to work well, but it is later in L253 discarded without providing enough details.\nThe experiments later (4.3) on compared to the simple set approach.\nWhy does the experiments now focus solely on this baseline?\nThe claim of using human help only when needed is evaluated in one single experiments.\nI wonder why this is not reported for all conducted experiments, since this is a core claim of the paper?\n\nLastly, the limitation section is another weak part of the paper.\nIt is not a sophisticated discussion of the approach but rather a summary of future work.\nThis approach makes many assumptions that are scattered across the paper, e.g., \"unique correct candidate plans\".\nThese are strong assumptions and need to be discussed.\nSimilarly, the creation of the calibration dataset is what provides the coverage guarantees.\nThere is no discussion of the calibration dataset and how difficult the creation of such datasets is.\nThe limitations section requires a major revision to truly discuss the current shortcomings of KnowNo when it can be applied and when it fails.\n\n\nOther minor things are:\n- The paper introduces POMDPs but does not use the concept throughout the paper. Why are POMDP introduced and what is partially observable? \n- The variable C is overloaded and used as the prediction set and the answer in the MCQA.\n- The variable s is used to denote the state and the score \n- Some notations are not introduced, e.g., Eq. 4 t\\in[T] (not even a proper interval definition) or \\hat{f}(\\tilde{x}^t))_y? \n- The package cite would help to avoid elongated citation number lists", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "PAkHlCRln5", "reviewer": "Reviewer_aKea", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper builds on the theory of conformal prediction to measure and align the uncertainty of LLM planners, asking for help when needed. The proposed method provides theoretical guarantees on task completion while minimizing required help. The method does not require fine-tuning the large language model. Instead, it generates multiple-choice questions and uses the estimated uncertainty to choose the suitable action.\nThe paper reports evaluation in situations and the real world, demonstrating the method's efficacy.", "summary_of_recommendation": "We recommend accepting the paper for publication. The proposed method is novel, and the results in simulation and real-world are convincing.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "No major questions", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper is well written, and the presentation is clear enough for the theoretical part.\n2. To the best of our knowledge, using conformal prediction to estimate uncertainty to improve LLM-based planners for robotics is novel.\n3. Sufficient experiments in the real world.\n\nWeaknesses:\n1. As mentioned in the limitation subsection, the method assumes a perfect textual description of the scene. The method considers only epistemic uncertainty while neglecting epistemic uncertainty coming from data. It would be interesting to discuss ways to estimate the observation's uncertainty and how to handle it.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "5qPpjG_6JP3", "reviewer": "Reviewer_4oDG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "LLMs have been used for robot planning in recent literature. This work is motivated by the \"hallucination\" issue of LLMs that tend to generate plans that are plausible in general but are incorrect or infeasible in practice. Looking into such issues, it's found that the current LLM-based planning systems lack the capability of leveraging its own confidence level in selecting a plan solution among those that are potentially feasible. This paper introduces a new algorithm and system called KnowNo that uses Conformal Prediction (CP) to align the uncertainty of LLM plans. More specifically, the LLMs generate a set of possible next-step actions, and ask help from human for decision making only if there are more than one above-CP-quantile solutions. KnowNo has been evaluated both in simulation and using a real robot in completing object rearrangement tasks. ", "summary_of_recommendation": "This research is timely and addresses an important problem in LLM-based robot planning. The developed approach makes some approximations that can potentially break the theoretical guarantees, where some clarification are needed. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See above. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "An important issue has been correctly identified in this research. The reviewer appreciates that this paper aims to address a fundamental challenge in LLM-based planning systems. The CP approach, as a statistical technique for computing confidence levels, is a well selected approach to address this issue. \n\nIf the reviewer understands it correctly, the single-step setting considers the tasks that require only one action (Section 3.2). If that's the case, are those tasks still pointing to \"planning\" problems? Some clarification would be appreciated. \n\nHow was the iid scenario dataset (with 400 instances) collected? It seems to be very important because the effectiveness of CP highly relies on the assumption that the labeled and unlabeled data instances are \"exchangeable\". Since the calibration dataset was collected before the execution time, how is the exchangeability guaranteed? The reviewer looks for a clear response to this question. \n\nContinuing the above comment, there's the statement that \"... to ensure the desired probability of coverage for each new ztest, one needs a fresh calibration set. But, in practice, we only calibrate once with a fixed set.\"  The reviewer is not sure how to interpret this statement. Does it mean that the whole algorithm loses the theoretical foundation in this research? Some elaborations about the gap between the fixed calibration set and the desired fresh set would be helpful. \n\nAnother concern is that in this research, human models are greatly simplified, which negatively affects the applicability of the developed approach in the real world. For instance, the human is assumed to be always available and equipped with complete knowledge, and no difference is assumed among different types of help from people (cost, accuracy, etc). While it's not the focus of this research, some discussions about human modeling would be nice. \n\nThere should be more advanced ways of using the calibrated confidence level of task plans. The current way of using it is quite simple -- if there are multiple plans above a quality threshold, the robot asks human to make the selection. Many other factors could be considered (including those about humans in the last comment) for better leveraging the uncertainty analysis -- some discussions about that would be nice. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines large language model (LLM) planning with uncertainty calibration to enable a robot to ask clarifying questions of humans. The paper builds on incorporating conformal predictions into the LLM planning and provides statistical guarantees on task completion. When it is unclear which option to follow, the robot is able to ask a human to clarify. For example, picking up a drink could mean picking up a coke or lemonade. A human can clarify this. Overall, the paper is of high quality with results in both simulation and on a real robot.
Decision: Accept (Oral)