Contrast Sets for Evaluating Language-Guided Robot Policies
Authors: Abrar Anwar
Rohan Gupta
Jesse Thomason
Venue: CoRL 2024 (2024)
Keywords: Evaluation
Language-guided robots
TL;DR: we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances for better evaluating language-guided robots.

Abstract: Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use contrast sets to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.

Contrast Sets for Evaluating Language-Guided Robot Policies
Abrar Anwar', Rohan Gupta', Jesse Thomason University Southern California (abraranw, rgupta96, jessetho) edu
Abstract: Robot evaluations in language-guides real world settings are time- consuming and often sample only small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an ap- proach to make small, but specific, perturbations otherwise independent, iden- tically distributed (i.id.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test per- formance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set pertur- bations to characterize policies at reduced experimenter effort in both a simulated manipulation task and physical robot saiion-and-languaae navigation task. We encourage the use of contrast set evaluations as more informative alternative small scale, i.idd. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.
Keywords: Evaluation, Language-guided Robots
Introduction
Language can be used for providing guidance on tasks like high-level task planning [1, 2], robot manipulation [3, 4], and visual navigation [5]. Robots are deployed in environments with many objects, and the space language commands robot can execute grows combinatorially with scene complexity. As such, these robots are often trained on large datasets that can specify hundreds of tasks in different environments. Evaluating such a robot on the large domain of its training set is impractical, especially as one typically needs to evaluate various policies. In simulation, researchers are able to evaluate their language-guice guided policies on robust, i.i.d. evaluation sets. However, since evaluation on physical robots is difficult, and experimenters usually demonstrate a robot's capabili- ties on small subset of tasks, falling short of the i.i.d. evaluation framework typical in simulation. In this work, we focus on evaluating language-guie guided robot policies efficiently so that experimenters can explore the large space possible instructions with less work.
Simulation is commonly used to evaluate language-guide policies [6. 7, 8, 5]. After training policy on various tasks, the policy is evaluated on large number of predefined test instances. Since simulations are typically insufficient for truly understanding policy's 's real-world performance [9, 10], and despite correlations between simulation and reality [11], there is need for an evaluation framework to systematically evaluate language- guided robot policies in the physical world. Since the space of possible robot behaviors in different scenes large, these approaches must also be efficient with respect to experimenter effort.
Consider manipulation and navigation tasks where a robot follows natural language instructions. To evaluate manipulation task, an experimenter has to move tabletop items to modify a scene, which takes experimenter effort. In navigation tasks, evaluations are trickier because the environment itself should vary between instances. Changing the environment in navigational settings often means moving furmiture or adding new large objects, which is labor-intensive. Additionally, when language is involved, these objects must also be semantically relevant. As a consequence, it is difficult to evaluate the performance of a language-guit guided policy scale due experimenter effort.
These authors contributed equally to this work.
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

Standard Test Set Evaluation
Contrast Sets
Contrast Set Evaluation
Extril Domain
Ftot inutance (4, 5,
Comtrast
perturbatioas of a test Instance along various exes
L14 MODEE
Apply perturiaations sequentially given test instance to cheaply cstiimat robot performance
Porturbations ancs
Sanguago di,
behavior
random
AL
Zaro cnit
perturb
sumple
Zero cost
05
AS
Minimal cost
Langunge
ASB
Minimal cost
Lamguage
Language
Evaluation with 1.dd sampling is expensive for experimenters!
Smaller pertarbations are cheap and estimate the original test set!
Figure I: Overview. Left: In standard test set evaluation, test set is i.i.d. random sampled cover the domain of possible language, scene and behaviors that a robot can execute. It can be expensive to reset the scene each new test instance during experiments. Middle: In this work, we design contrast sets [12] for language-guided robot evaluation, comprising perturbation strategies based on the language, scene, and expected behavior of the robot. Right: The proposed contrast set evaluation allows experimenters to efficiently evaluate neighborhoods around original test instances.
Contrast sets [12] use deliberate perturbations of i.i.d. test sets to enable better evaluation of a ma- chine learning model. We take inspiration from contrast sets and propose perturbation functions to systematically and efficiently evaluate the space of robot test instances. In this work, we build framework for contrast set- set-based perturbation strategies for evaluating aaguaae-guiiedd robot poli- cies. We apply perturbations on test set instances in the language, scene, and behavior axes, which allows for a lower cost evaluation. This framework strikes balance between expensive i.i.d. sam- pling of test instances and cheaper test instance perturbations. In particular, we
design contrast set-based perturbation strategies for exploring the test domain; demonstrate how policy performance across different types of perturbations lend insight into axes of policy robustness and brittlenes;;
design a simulated manipulation task and real-world and demonstrate that our contrast set evaluation efficiently estimates policy performance.
ann-lang navigation task
2 Background and Related Work
Past work in machine learning model evaluation has used perturbations as a method to probe model performance; however, evaluation in robotics typically focuses small number of pre-defined tasks. In this section, we discuss contrast sets from NLP, and its relevance to evaluation for robotics.
Evaluation in Machine Learning. A large, sampled i.i.d. test set may not capture the span of expected situations a machine learning model could encounter in the real world. To address this, researchers have designed diittriibiiiitooonn evaluation techniques in the vision [13, 14, 15, 16] and NLP [17, 18, 12] communities. In computer vision, perturbations of images have been used to generate counterfactual examples to test model [19, 20, 21, 22, 23). These approaches allow experimenters to stress test their models and have confidence in their model during deployment. However, in robotics, testing requires physical deployment and takes considerable efforts to compute such metrics. In this work, we focus on designing contrast sets for language-
robot policies.
Evaluation of Robot Policies. Simulation is a common way to train and evaluate robot poli- cies [24, 25, 26]. These simulated environments are often used for evaluating the performance of a real-robot system [10, 9, 27, 28] by recreating a simulated counterpart to a real environment, but show ineffective direct sim2real performance without domain randomization or real-world fine- tuning strategies. There exist correlations between simulation and real-world performance even if they do not exactly match [29, 11]: however there are no guarantees about real-world performance. These works also pre-define a set number of tasks in simulation; but it is not scalable to engineer simulators for every new task. Other recent work consider how to evaluate language-guiie guided robots in the real world such as evaluating L.LM- task planners [30] or providing bounds on policy

---

performance [31]. Additionally, RL frameworks such as autonomous RL [32] and others [33, 34] focus on minimizing the number of human interventions during deployment. Contemporary work have begun to investigate how different visual or linguistic attributes impact evaluation performance [35, 36, 37] and efficient data collection [38]. We find our work complementary to these, and focus on evaluating the space of possible test instructions and scenes in an efficient manner.
Contrast sets in NLP. Contrast sets are perturbed variants of the test set that help characterize the decision boundary of a classification model. They are constructed by perturbing the input text and/or the output label. For example, robustness to sarcasm could append "Yeah, right!" to the text of positive review, indicating a change from a positive label to a negative label. We design contrast sets for robotics that alter accompanying language instructions of test instances and potentially the expected behavior.
sentiment classification task, perturbation to test model
Contrast sets with vision. In the NVLR2 [39] visual reasoning task, contrast sets are formed by replacing an image with one that contains a minimal change that may alter the answer to an accompanying question. For example, for a test that asks whether an image contains "two chow dogs facing each other", an image perturbation finds one that is semantically similar-two dogs are facing each other-but the dogs are of different breeds, thus the label is flipped. We design contrast sets for robotics that alter the start state of instances and potentially the expected behavior.
3 Problem Statement and Notation
Language-guiden policies can be learned from large-scale collected data, then deployed to control physical robots. Systematic, efficient approaches to evaluate learned policies controlling physical robots will facilitate understanding how effectively those policies can cover the domain of test in- stances. In this paper, introduce evaluation strategy inspired by contrast sets [12] estimate a given policy's 's performance on a fixed evaluation set measured by given metric while minimizing the physical cost of setting the initial conditions of each evaluation instance.
The space of discrete language instructions is notoriously large, so these policies are typically eval- uated in simulation over thousands of instructions. Several works have focused on correlating sim- ulation performance to real world performance on a handful of instructions for tuning the sim2real gap [11] on evaluating image-based navigation policies in Airbnbs rented homes [40, 28]. Tai- loring simulations for new skills or renting dozens of environments are not scalable paradigms as experimenters must spend dozens of hours to cover situations policies may encounter.
Evaluation sets. We formalize the problem of evaluating Language-guides robot policies in domain of test instances X' and range of expected behaviors ). An evaluation set X Y:X Y,,YC. y is composed of instances, the initial conditions (1, E X faced by a policy and the desired outcome b Y, characterized by the language instruction 1, the starting scene 8, and the expected optimal behavior b. For notational simplicity, we specify a language- guided policy I which takes in initial conditions I, 8 and produces behaviors b rather than that of an iterated state-to-action definition: f(l,s) + b. To evaluate a language- -rnggee-guddd robot policy f. experimenters sample an evaluation set X {X,,..., [X, ~ X} with associated behaviors Y = (human (X.) Y;; X, E X). We assume that the sampled evaluation set X is representative of the test manifold defined by X. A trained, language-guide robot policy f 0 x X Y is evaluated over X to produce Y. against which performance metric M Y) is calculated.
Evaluation Strategies. An evaluation strategy sequentially samples or modifies test instances X into sequence of test instances to be executed, with experimenter intervention to set up each subsequent starting condition. A standard evaluation strategy I(X) = (x, E X|Vi e I,... n) simply converts the evaluation set into shuffled sequence of test instances. Let P be a set of perturbations functions. A strategy I' can construct larger test set by applying perturbations ((X) (ō(z)Jæ, E O E P, Vi E II) In this paper, we highlight the comparative ad- vantages of such a perturbation beturbbation-based evaluation strategy.
Cost and metries. To calculate the cost of a series of evaluations, we define an evaluation cost C(I(X)). This cost will give us insight to how much experimenter effort a given evaluation strategy takes. The purpose of a test set is to estimate the value of given metric (()(())), Y) that is an indicator of the robot's 's performance. The metric M and evaluation cost C are general terms and are chosen depending on the setting. The metric could be reward in an environment, success rate,
3

---

success-weighted path length, or the distance from goal, while the evaluation cost metric could be distance of objects moved, time taken, or energy used reset the scene.
The i.i.d. random sampling strategy [(X) is typically ideal for estimating an evaluation metric for given policy. However, the cost over this strategy C(I(X)) likely exceeds any given cost budget K, because i.i.d. sampling of test instances is not aware of the cost of changing instances. The goal of our work is design strategy [((X over the evaluation set using cost-effective perturbations P such that (X)) < of evaluations bounded by the cost budget K. This formulation means that the size of the contrast set is often greater than the size of the standard evaluation [ (X) > [[()]. The procedure for
()) while (I K (X)) & (I())), where I' x (X) is the sequence
applying set of perturbations P given cost budget K is shown Algorithm in the Appendix.
In this work, we consider the problem of efficiently evaluating ferent instructions, scenes and behaviors. As such, we do not foeus language-conditioned pick and place tasks as the scope of perturbations is limited in these environments. We consider set- tings where it is possible compose language instructions define innumerable numbers of robot behaviors. Specifically, we use the SanguageTTable [41] and VLN-CE [42] tasks, as the language instructions in these environments specifically relate to how the actions are taken.
gegguage-guider robots across dif-
4 Contrast Set Evaluation Strategies
We adapt contrast set perturbations to design strategy r(X) that perturbs an evaluation set in the context of aunuagg-gguidd, visual sequence-to-sequence problems. We then instantiate these perturbations in simulated manipulation task and a physical rision-and-language navigation task.
Contrast sets for robots. Each test instance is characterized by the language instruction I, the scene and the expected behavior b (Figure 1). Perturbations to the language and scene can lead to changes in the expected behavior, we define our own scene and language perturbation functions that may or may not modify the expected behavior. We define four types of perturbation functions, denoted by the symbol A and a letter for the axes they modify.
(x(x) perturbs the language instruction such that the expected behavior is the same.
ALB(a (x)) perturbs the language instruction such that the expected behavior is different.
(x: perturbs the environment such that the expected behavior is the same.
AS)(r perturbs the environment such that the expected behavior is different.
Instantiated perturbation functions P depend on the tasks being evaluated. We measure cost with respect to modifying the environment, so language-based perturbations do not add any additional physical cost. We do not include the cost of resetting the robot position as it is fixed across evalua- tions, and in some cases can be automated. Our work investigates whether contrast sets are able to estimate the sample mean performance of a standard evaluation set with a similar or lower cost.
Contrast sets vs i.id. evaluation sets. In comparison to a standard evaluation I(X), where an experimenter sequentially executes i.i.d. random test instances with no consideration of the cost, contrast set evaluation allows the experimenter to explore the test domain and cover it efficiently.
Often, when researchers demonstrate the capabilities of their policy, they do not construct an i.i.d. evaluation set, but instead only apply perturbations. These demonstrations are typically done to cheaply evaluate their robot in limited domain. However, because those test instances and pertur- bations are not i.i.d., those works are not properly evaluating their robot systems. Since our work assumes access to an i.idd test set and then applies perturbations in sequence, contrast sets for robots strikes a balance between saving experimenter effort and properly evaluating robot policy. Through more systematic evaluations in the test domain, contrast sets enable better coverage of the test domain, accurate performance estimates, and insights on a policy's sensitivity to perturbations.
Table Simulator Experiments
Tanguage-Table [41] is a multi-task language-conditioned control benchmark that spans 696 unique task conditions across five categories, with each specified in language in dozens of ways. The general task for a manipulator to push blocks of various shapes and colors to specific relative or absolute positions based on a language instruction, as visualized in Figure 2. It is infeasible to evaluate all of

---

Standard
separate blocks
relative block-to-baooc
block to block
relative to block
block absolute location
Evaluation
pull spart from blue moon
green right of the green cube
put the green to the green cube
alightly green cube up and right
pul green upper right corner
Contrast Set
Aßeer
Evaluation
nO reset
perturb
Milt
new
ASIl:
instance
directoos
block
push
push
moon the green star
upwards and right
down and left
the green alar
diagonally
diagonally
Table Figure 2: Language-Tabl Rollouts. In the Language- Table simualtor [41], we sample an evalu- ation set of 250 test instances that sequentially evaluated. A test instance sampled from one of five task types which manipulate blocks according to a task definition. The standard evaluation requires i.i.d. random sampling instructions and scenes, which accumulate more effort for the exper- imenter. Contrast set evaluation allows experimenters to perturb sampled test instances by making minimal changes after each execution, leading to less work for the experimenter.
these test instances in physical environment, especially there are multiple checkpoints or models to evaluate, making it important study how to efficiently evaluate large task domains.
Experiment Design
We compare standard evaluation set X against strategies (XX that various perturbation sets P. Over three seeds, we sample an evaluation set from the simulator consisting of 250 test instances sampled across five different task categories, from which
various evaluation strategies.
Using perturbation functions P. we perturb these test instances with low cost. We believe it would normally be infeasible to reset the scene after each evaluation, between perturbed test instances, we do not reset the environment. However, if we applied perturbations that do not modify the expected goal of the robot, perturbed test instance may already be in success state. Therefore, we do not use the AL and A functions and define two ALB perturbations and two shown in Figure 2. The test instructions typically involve source block that needs to be moved, a target block that a block needs to move relative to, or a direction such as "top" "right". The ALB, perturbation swaps the target and source block referring expressions in the instruction. ALB, modifies the instruction such that any directions or positions are flipped, for example upper lefi is changed to lower right. ASB, moves the target block to new location, while ASB2 moves the source block to a new location. Given a pre-defined test set, we design various contrast sets from these perturbation strategies determine their impact on effort.
Perturbations.
ASB perturbations
Metries. We use an optimal planner to compute the the success weighted by path length (SPL) as the metric M (-) such that suboptimal trajectories penalized. As a proxy for experimenter effort in resetting a scene, we define the cost C(.) as the distance (in meters) objects are moved in a scene.
5.2 Results and Discussion
Figure 3 summarizes the differences in policy performance on contrast sets that target particular language- and scene-based perturbations versus the original test set and shows that contrast set evaluation does not compromise the estimate the test set M(.) achieved via Standard Evaluation.
Contrast sets show the policy's sensitivity to perturbations. Since the perturbations are concrete and specific, they can show policy's sensitivity to the language and scene axes. shown in Figure 3, find that the tanguage-basee perturbation ALB, is notably below the mean SPL for the
5

---

0.35
Foll Controst Set
Standard Evaluation
Limtted
0.30
0.2
Scone Parturbation
Language Perturbations
Language Pertarbations
0.20
Moan
Absaluta
0.0
0.15
0.10
-0.2
0.00
Original
ASR
-0.3
Nettorbation
Cumulativn Cost moved)
Figure 3: Left: A key insight offered by contrast set evaluations is probing the strengths and weak- nesses of learned policy. The mean success-weightte path length (SPL) achieved over the full test set may compare average policy performances, but here we observe additional robustness instruc- tion source and target switches and source block starting position (ALB,,AS B2) but brittleness to direction word inversion (ALB2), providing insights for training and deployment. Right: Compar- ison of evaluation strategies' absolute estimation eтrоr the SPL of the entire test set as a function of the cumulative cost in distance blocks are moved during standard evaluation is 281, achieving the horizontal error line at 0.0, and we cap cost at 300, though additional perturbation instances are possible for some strategies. All perturbation strategies achieve better test set SPL estimates than a Limited Intervention baseline.
resets. The maximum cost of the
policy. ALB₂ simply swaps the directions in text to be the opposite. We took a test instruction of moving a block to the right changed the direction to left with ALB2, and we found that the policy would continue to move the block to the right This reduction in performance indicates that the trained model may be overfit to direction some cases. By contrast, ASB2' 's higher SPL relative to the contrast set indicates the policy is robust to source block locations. Intuitively, this result shows that perturbations help qualitatively characterize different regions of the test manifold.
Limited interventions are not predictive of SPL. Standard evaluation i.i.d. samples new test in- stances. However, practice, sometimes experimenters simply execute new language instructions in given scene until they cannot anymore, which allows them to reduce the amount of effort re- quired by minimally intervening to reset their scene. We explore a limited intervention evaluation strategy in which new language instructions are sampled from the evaluation set sequentially with- out resetting the scene. This experiment meant to be a lower bound in cost, where new language instructions and behaviors are sampled, but the reset costs are is minimized by not resetting the scene unless planner determines an instruction infeasible. As shown Figure 3, 250 trials of this strategy yields much lower cost than all of the other strategies, but poorly estimates the test SPL.
Contrast sets improve predictive performance and provide cost-effective approach exe- cuting more trials. Figure 3 shows that language-busee perturbations underestimate the SPL of the evaluation set while scene-based perturbations overestimates it. Using all four perturbation func- tions together yields the best predictive performance, as it converges closest to the full evaluation set performance, As shown in Figure in the Appendix, the standard evaluation incurs cost of 281 me- ters over 250 trials. In comparison, the language-basee contrast set reaches around 400 trials for the same cost but under-estimates test performance. The combined scene and language- based contrast set allows for 380 trials without significant under- or over-estimation of the test performance.
6 VLN-CE Evaluation on a Physical Robot
Vision-ande language navigation in continuous environments [42] (VLN-CE) is a task where agent follows fine- graigna language instructions household setting. We deploy a VLN-CE agent to control real world Locobot [43]. By evaluating with contrast sets, are able draw insights about policy sensitivity and estimate the full test performance with lower cost to the experimenter.
6.1 Experiment Design
We design pseudo studio apartment environment populated with furniture similar to categories in simulation. To ensure ecological validity of test instances, we recruited five participants to design

---

Test Insiaace
More All
Vese ABl Objach
Evaluation
All Objecb
All Objech
"Gu pass the laup and the plants, keep going and stop when you reech the bookshelf"
"Go by couch, right and the sighnsand"
lure
go to had, thei tum right and go the table and stop.
"Go to the couch, and hun to
around and go to the bed and stop."
your lett and will see
bookshelf Go
stop.
Tost Instance
Test Instance AL
Test Instance İ.ALB
Test Enstance 1,AS
That Instance LSA
Evaluation
Change Warding
Change Finsl
Coath
God
"Go pass by the lamp and the plants, keep going and the bookshelf"
"Move post the lamp and the until
"Coo pass by the lump and the plants, like right, keep
"Go pasa by the lump and the plants going and stop
"Go by the lamp and the keep going when bookshelf"
Sanksbeff
stop when
slp
Figure 4: VLN-CE Robot Rollouts. The standard evaluation of i.id. random sampling scenes requires scenes to be shuffled around drastically. Intuitively, contrast sets allow experimenters to cheaply perturb sampled test instances to find ecologically valid samples to evaluate.
five furniture setups, each of which they annotated with a navigation instruction as shown in Figure 4. Each test instance x and outcome y is defined by the scene based on the furniture arrangement and the robot's pose, and the expected behavior of the robot, described by the language instruction I consisting of two subgoals to be reached. More details on this protocol for ensuring ecological validity and model training can be found in Appendix C.
Perturbations. We define four perturbation functionn as shown in Figure 4. We define language- only perturbations AL to simply change the wording of the instruction but preserving the meaning. If language instruction tells the policy to go to bed then to the couch, then this perturbation simply rephrases the instruction. ALB changes the final goal the text instruction, so the first half of the robot's behavior to the first goal is the same, but the latter half changes. AS moves object around in the scene such that the expected trajectory the robot is still the same. For example, passive objects the robot is not meant interact with may change positions long they do not change the interpretation of either language subgoal. Lastly, ASB moves an object around in the scene such that the expected trajectory of the robot is different. This perturbation either moves the goal object such that the trajectory must be different, or an object is moved in front of the robot to impede its originally-intended trajectory.
Metrics. We define the cost metric as the distance objects were moved in the scene. The metric the average progress toward two subgoals: 50% for reaching the first and 100% for reaching both. We evaluate each test instance three times. Four perturbation strategies applied to five original test instances yield 20 new instances Executing these 20 test instances and the original test instances three times results in a strategy set size of [(X) = 75.
6.2 Results and Discussion
Modifying the scene, robot malfunctions, and maintenance all add to the cost of evaluating robots. We find that contrast sets reduce the effort required for evaluating robots in the real world.
Contrast set perturbations enable more experiments for less work. The contrast set consists the Appendix.the average cost per instance in
of 4 perturbation functions. Based on Figure the standard evaluation set is 8.8 meters, while generating 4 perturbed trials from a single i.i.d. test instance costs about I meter. With 5 i.i.d. test instances from the standard evaluation set, the additional cost of meter from perturbations allows one to execute 4 5 = 20 more trials.
Contrast sets estimate the full test set with less cost. Figure 5 demonstrates that contrast set evaluation converges to a similar success rate as standard evaluation while requiring less effort, indicating that this evaluation allows experimenters efficiently measure policy's performance.
Contrast sets can provide insights into policy's linguistic and visual understanding. Similar to results in the simulation results, our specific perturbations can show a policy's 's sensitivity to specific
7

---

Full Contrast Set
Standard Evaluation
Evalustion
0.4
0.4
Moain Progresa
Absoluia Proqress To
0.2
0.0
0.0
Original
AL
ALB
A5
ASB
0 10 20 30 40
50
Perturbation
Cumulative Cost (Distance moved)
Figure 5: Left:: Contrast set evaluation probes the strengths of trained VLN-CE model to a physical robot. We observe that the policy robust to changes the final goal instruction (ALB) and physical changes to the goal itself (ASB) as depicted by the 13% higher performance over the full contrast Right: Average cumulative progress to goal error (and cumulative std. dev.) cumulative cost. The contrast set evaluation quickly reaches a nearly accurate estimate of the final test set progress goal, showcasing the potential reduce experimenter costs dramatically by exploring neighborhoods of contrast around test instances.
axes. We find that ÅL and AS is lower in performance than that of the full contrast set, especially compared to the higher performing contrast sets. For example, we found that instruction "Go pass by the lamp and the plants, keep going and stop when you reach the bookshelf" performed with an average progress goal of 100% over three runs, while the reworded version in the same setting achieved 33%. We found this trend to continue, and it directly highlights potential issues the lan- guage encoder, the language representations are likely well aligned. Though perturbations for this experiment are relatively general, more specific perturbations can highlight regions of the test domain where policy is particularly effective and where it may need improvement.
7 Conclusion and Limitations
A language-guided robot can be tasked with potentially thousands of tasks across arbitrary envi- ronments. evaluate such robot, experimenters ideally construct i.i.d. test sets that effectively cover the domain of possible test instances. However, creating these sets often require extensive rearrangement of the environment, increasing experimenter effort. In this work, we propose contrast sets as approach for evaluating language-guide robot policies. We find that evaluating these contrast sets provides insights into policy robustness and sensitivity. Additionally, contrast sets are able to estimate the full evaluation set performance while maintaining low experimenter cost. We conducted simulated manipulation and real-world ssiion-and-languaa navigation experiments and found that contrast sets enable experimenters to more evaluations with less effort. We argue that contrast set evaluations offer higher fidelity than small-scale, real robot demonstrations while not requiring the industry-level resources for large-scale, deployed evaluations.
While we focused primarily on environments where language instructions specify greater details on how the robot should interact with the environment, we intend to investigate larger manipulation- focused datasets such DROID [44] and OXE [45]. Additionally, we want to emphasize ecologi- cally valid language instructions that perturbations may deviate from, which could involve new scene perturbations (e.g., new object instances). language perturbations (e.g.. changing cultural contexts), or different cost types (e.g., labor costs). We assumed the existence of a test set that is representative of the test manifold, and that contrast set perturbations will generate instances within the domain. Unlike other works, we do not aim to estimate performance outside this domain. Typically, test sets are not well-defined advance and future work can explore which perturbations can allow experi- menters to efficiently evaluate more open-ended robot systems. Now that work has shown the utility perturbation strategies for efficiently evaluating robot policies, future work can consider automatic approaches to selecting which perturbations to apply and for generating new test instances for experimenters, potentially using large language models.

---

Acknowledgments
This work was supported in part by grant from the Army Research Lab (ARL) Army Al Innova- tions Institute (A212), award number 1NF-2333-2000100 Rohan Gupta was supported by USC Provost's Undergrad Research Fellowship during portion of this work. The authors would like to thank Elle Szabo for building the early infrastructure for the physical robot experiments.
References
[1] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do As I Can and Not As I Say: Grounding Language in Robotic Affordances. Conference on Robot Learning (CoRL), 2022.
[2] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. International Conference on Robotics and Automation (ICRA), 2023.
[3] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. Conference on Robot Learning (CoRL), 2022.
[4] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation. Conference on Robot Learning (CoRL), 2023.
[5] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-languaal navigation: Interpreting grouudde naviga- tion instructions in real environments. Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
[6] K. Zheng, X. Chen, O. C. Jenkins, and X. Wang. VLMbench: A Compositional Benchmark for Vision-and and-LLaguage Manipulation. Conference on Neural Information Processing Systems (NeurIPS), 2022.
[7] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, Zettleoyer, and D. Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[8] O. Mees, L. Hermann, E. Rosete-Beat and W. Burgard. CALVIN: Benchmark for Language- Cottuagg-CCdiiioeeed Policy Learning for Long-Horizon Robot Manipulation Tasks. 1EEE Robotics and Automation Letters (RA-L), 2022.
[9] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for vision-and-language navigation. Conference on Robot Learning (CoRL), 2021.
[10] M. Deitke, W. Han, A. Herrasti, A. Kembhavi. E. Kolve, R. Mottaghi, Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, L. Weihs, M. Yatskar, and A. Farhadi. RoboTHOR: An Open Simulation-to Re-l Embodied AI Platform. Conference on Computer Vision and Pat- tern Recognition (CVPR), 2020.
[11] X. Li, K. Hsu, Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kir- mani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint 24554:200.044 2024.
[12] M. Gardner, Y. Artzi, V. Basmov, J. Berant, B. Bogin, S. Chen, P. Dasigi. D. Dua, Y. Elazar, A. Gottumukkala, N. Gupta, H. Hajishirzi. G. Ilharco, D. Khashabi, K. Lin, J. Liu, N. F. Liu, P. Mulcaire, Q. Ning, S. Singh, N. A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, A. Zhang, and B. Zhou. Evaluating models' local decision boundaries via contrast sets. Find- ings of Empirical Methods in Natural Language Processing (EMNLP Findings), 2020.
[13] D. Wang, N. Ding, P. Li, and H.-T. Zheng. CLINE: Contrastive Learning with Semantic Neg- ative Examples for Natural Language Understanding. Association for Computational Linguis- ties (ACL), 2021.

---

[14] D. Hendrycks. X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song. Pretrained trans- formers improve out-of-distribution robustness. Association for Computational Linguisties (ACL.,, 2020.
[15] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to ima- genet? International Conference on Machine Learning (ICML). 2019.
[16] D. Hendrycks and 'T. Dietterich. Benchmarking neural network robustness common cor- ruptions and perturbations. International Conference on Learning Representations (ICLR), 2019.
[17] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:221 2022.
[18] Y. Chang, X. Wang, J. Wang. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang. al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology (TIST), 2024.
[19] M. Zemni, M. Chen, É. Zablocki, H. Ben- Younes, P. Pérez, and M. Cord. OCTET: Object- aware Counterfactual Explanations. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[20] G. Jeanneret, Simon, and F. Jurie. Adversarial counterfactual visual explanations. Confer- ence on Computer Vision and Puttern Recognition (CVPR), 2023.
[21] G. Jeanneret, L.. Simon, and F. Jurie. Diffusion models for counterfactual explanations. Asian Conference Computer Vision (ACCV), 2022.
[22] A. Sauer and A. Geiger. Counterfactual generative networks. International Conference Learning Representations (ICLR), 2021.
[23] V. Prabhu, S. Yenamandra, P. Chattopadhyay, and J. Hoffman. LANCE: Stress-testing Vi- sual Models by Generating Language-guide Counterfactual Images. Conference Neural Information Processing Systems (NeurIPS), 2024.
[24] Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots arXiv preprint arXiv:2406.0255 02523,, 2024.
[25] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. Conference on Robot Learning (CoRI.). 2021.
[26] James, Ma, D. R. Arrojo, and A. Davison. RLBench: learning environment. IEEE Robotics and Automation Letters (RA-L), 2020.
The robot learning benchmark &
[27] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, F. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Perfor- mance? IEEE Robotics and Automation Letters (RA-L), 2020.
[28] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects in the real world. Science Robotics, 2023.
[29] W. Pumacay, L. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. THE COLOSSEUM:
A Benchmark for Evaluating Generalization for Robotic Manipulation. Robotics: Science and Systems (RSS), 2024.
[30] Z. Hu, F. Lucchetti, C. Schlesinger, Y. Saxena, A. Freeman, S. Modak, A. Guha, and J. Biswas. Deploying and Evaluating LLMS to Program Service Mobile Robots. IEEE Robotics and Automation Letters (RA-L), 2024.
[31] J. A. Vincent, H. Nishimura, M. Itkina, P. Shah, M. Schwager, and T. Kollar. How General- izable My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation. arXiv preprint arXiv: 240::2405.044 05439, 2024.
10

---

[32] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, S. Levine, and C. Finn. Autonomous reinforcement learning: Formalism and benchmarking. International Conference on Learning Representations (ICIR), 2022.
[33] A. Chen, A. Sharma, S. Levine, and C. Finn. You only live once:
Single-life reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[34] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. International Conference on Robotics and Aulomation (ICRA), 2021.
[35] A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the generalization gap imitation learning for visual robotic manipulation. International Conference on Robotics and Automation (ICRA), 2024.
[36] S. Jin, R. Wang. M. Zahid, and T. Pokorny. Video Transformers under Occlusion: How Physics and Background Attributes Impact Large Models for Robotic Manipulation. arXiv preprint, 2023.
[37] A. Parekh, N. Vitsakis, A. Suglia, and I. Konstas. Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks. arXiv preprint arXiv:2407.03967, 2024.
[38] Gao, A. Xie, T. Xiao, C. Finn, and D. Sadigh. Efficient Data Collection for Robotic Ma- nipulation via Compositional Generalization. Proceedings of Robotics: Science and Systems (RSS), 2024.
[39] A. Suhr, S. Zhou, Zhang, Zhang, H. Bai, and Y. Artzi A corpus for reasoning about natural language grounded in photographs. Association for Computational Linguistics (ACL), 2019.
[40] Y. Min, Y.HH. H. Tsai, W. Ding, A. Farhadi, R. Salakhutdinov, Y. Bisk, and J. Zhang- Self-supervised object goal navigation with in-situ finetuning. International Conference Intelligent Robots and Systems (IROS), 2023.
[41] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters (RA-L), 2023.
[42] J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. European Conference on Computer Vision (ECCV), 2020.
[43] Trossen Robotics LoCoBot, 2024.
httts::/www.trrossee
com/
locobot wowbbbot-veeviee....ooo asw...
[44] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, T. Miller, J. Wu, Belkhale, S. Dass, H. Ha, A. Jain, A. Lee. Y. Lee, M. Memmel, Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mer- cat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, D. A. Herrera, M. Heo, K. Hsu, J. Hu, D. Jackson, C. Le. Y. Li, K. Lin, R. Lin, Z. Ma. A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O'Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang. Y. Wu, A. Xie, J. Yang, Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, Gupta, D. Jayara- man, J. J. Lim, J. Malik, R. Martin-Martin, S. Ramamoorthy, D. Sadigh. S. Song, J. Wu, M. C. Yip. Y. Zhu, T. Kollar, S. Levine. and C. Finn. DROID: Large-Scale In-The-Wild Robot Manipulation Dataset. Robotics: Science and Systems (RSS), 2024.
[45] O. X.-F. Collaboration, A. O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi,
11

---

A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid,
B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Iehter, C. Lu, Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalash- nikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang. G. Wang, H. Su, H.-S. Fang. H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Fu- ruta, H. Walke, H. Fang. H. Ha, I. Mordatch, I. Radosavovic, Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Bohg. J. Bingham, J. Wu, J. Gao, J. Hu,
J. Wu, J. Wu. J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang. J. Salvador, J. J. Lim, J. Han, K. Wang. K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne K. Oslund,
K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, I.. Pinto,
L. Fei-Fei, L. Tan, L.. J. Fan, L. Ott, L.. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, T. Miller, Yin, P. Wohlhart, P. Xu, D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart'in In-Mrrt'ie in, R. Baijal, Scalise, R. Hendrix. R. Lin, R. Qian, R. Zhang, R. Men- donca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, Bahl, S. Dass, S. Sonawani,
O. Bastani, R. Sanketi,
Song. S. Xu, S. Haldar, Karamcheti, Adebola, Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding. T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Van- houcke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang. X. Zhu X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang. Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang. Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. https::/ aawiiiiiivvv... orgg abs 2310.020000 68,64, 2023.
[46] A. Ku, P. Anderson, R. Patel, E. le, and J. Baldridge. Room-Across-Room: Multilingual Saiionannd-Language Navigation with Dense Spatiotemporal Grounding. Conference on Em- pirical Methods in Natural Language Processing (EMNLP),202
12

---

A Contrast Sets for Robotics
Overall, our work investigates whether contrast sets able estimate the sample mean perfor- mance of a standard evaluation set with a similar or lower cost. We do not make estimates of the true population mean of the performance (i.e. run the experiment a large number times so that the performance converges to single true mean). Instead, we focus sample mean performance comparisons because experimenters can realistically only estimate the sample mean and not the true population mean.
Algorithm Contrast set with perturbations
Input: Test set X, budget K, perturbation functions
2: Ctotal +- 0
3: for X do
4:
for d E P do
5:
C costost22'10 (2', 21) x',9 f()(x)) Ctotal & Geotal 8: if Ctotal K then
6:
Dr x is the resultant state
7:
End 10: end if
9:
11: end for
12:
end for
B Language-Table Simulator Experiments
We describe additional details left out of the main text on the Language-Taie Table simulator.
B.1
Model Details
For the policy, we use the pretrained FiLM-conditioned ResNet architecture that was trained using behavior cloning provided by the Language- Table repository [41]. We do not Language-Tabi Table's LAVA model as a pretrained model was not provided and requires 64 TPUv3 chips to train.
B.2 Additional Details
In this section, we describe how the cumulative cost plot in Figure was generated. Since we evaluated over three seeds and each experiment has different cost, we create bins at equal intervals from 0 to the max overall cost across all seeds, then aggregate the cumulative absolute SPL error and cumulative cost. Using this binning approach, we also compute the standard deviation of the error bounds.
B.3
Additional Results
Contrast sets allow for more evaluations with less cost. As depicted in Figure 6, the slopes of each type of perturbations determines how the cost scales compared to the number of evaluations. Limited interventions is clearly the lowest cost; however, we had found that does not estimate the evaluation set. All contrast set strategies have a higher slope than that of standard evaluation. For example, scene and language perturbations can execute nearly double the number of experiments compared to the standard evaluation given cost budget of 281.
C VLN-CE Evaluation on a Physical Robot
We a Locobot [43] robot to run vision-and aadd--an--aana and-language navigation in continuous environments [42] (VLN-CE) in the real world.
C.1 Model Details
13

---

We pretrain a policy for the robot on the VLN- CE task the Habitat simulator using the
0.6
Standand Evalustion
RxR training set [46]. We then use behav- ior cloning objective to finetune the simulation- trained model on small set of real world ex- amples using teacher-forcing. The policy uses discrete action space of forward, rotate left by 30 degrees, rotate right by 30 degrees, and used
Absoluto Progress Goal Eroor
Evaluation
0.4
0.2
0.0
scene arrangement in the training dataset, and this scene was not used during testing. We note that the furni- ture, especially larger items such as beds and couches, used during training and existed during training. However, the scene arrange- ments, which is key the task of VLN-CE, was ensured to be different.
stop. Only
-0.2
0
250
500
750
1000
1250
1500
1750
Cumulative Cost. (Time)
Figure 7: Average cumulative progress to goal vs cumulative cost as time to perturb the scene in seconds. We find that the results found in Figure and Section 6.2 when using time as the cost func- tion instead of the distance objects were moved still hold when switching to time to perturb the scene as the cost. This shows that we may be able to use various cost metrics to measure exper- imenter effort.
C.2 Experiment Design
We describe how we collected our test in- stances. We design pseudo studio apartment environment which populated with furniture similar to those found in simulation. To en- sure ecological validity test instances, we re- cruited five participants to design five furniture setups. They were instructed to ensure that the furniture arranged in any way they would prefer, defining the scene They then placed the robot and walked trajectory 6 they wanted the robot to execute while narrating natural language command I. A subset of the navigation instructions can be found in Figure 4. By using external participants to design our test instances, we hope to ensure that we, as experimenters, do not bias the collection of our test instances be easier than expected.
C.3 Additional Results
Contrast sets allow for more evaluations with less cost. As depicted in Figure 8, the slopes of each type of perturbations determines how the cost scales compared the number of evaluations. Though contrast set evaluation has a higher bound, given a cost budget of 35, contrast sets allow user to run nearly triple the number of trials for the same cost budget.
Number of evaluations
Standard Evaluation
800
Limited Interventions
Scene Perturbations
Language Perturbations
600
Scene & Language Perturbations
400
200
0
0
50
100
150
200
250
300
Cumulative sum of reset cost
Figure 6: Compared Figure 3, we separate the relationship between cost and error. Limited inter- ventions and language-only perturbations allows for more evaluations with less cost, and standard evaluation has the least number of evaluations for the cost. described the main text, scene and language pertubations finds good middle ground with more evaluations for less cost.
14

---

25
Number of evaluations
Standard Evaluation
20
Contrast Set Evaluation
15
10
0
10
20
30
40
50
Cumulative Cost (Distance moved)
Figure 8: We separate the relationship between cost and error in the real word VLN-CE experi- ments. We that the number of evaluations performed in the standard evaluation is relatively low compared to the contrast set evaluation. Contrast set evaluation allows the experimenter to execute more experiments compared standard evaluation. Though not every single experiment from the test set can be executed under cost budget of around 35 (blue dotted line), Figure 5 indicates that contrast set evaluation still estimates the test set.
Contrast sets also estimate the full test set while minimizing time reset the scene. Instead of using distance of objects moved during a scene reset as we did in the main text, we also investigate the time used reset the scene cost metric. We find similar results in Figure 7 which uses time as cost as we did in Figure 5 which uses distance of objects moved as cost. This is likely due to the nearly-linear relationship between time takes to move items in the scene and the distance they are moved.
15


Reviews:
[{"review_id": "aM7ufJ0Dql", "reviewer": "Reviewer_5QZG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper contributes a novel way to construct robot test sets by using contrast-based perturbation strategies to design test instances.", "summary_of_recommendation": "This paper provides a reasonable advancement to evaluation suites for language-guided robots. While I think it requires more technical delineation to be practically feasible, I would still be in favor of accepting the paper for its explicit focus on evaluation, an area that I feel is oft ignored in learning-based robotics.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* How did the authors come up with the 4 specific perturbations to be tested, and what is the justification/reasoning for why they believe this set to be comprehensive of the full space of perturbations that may exist in language-based evaluation?\n* As stated above, it's very unlikely that real human language specifications will be as highly descriptive as the two domains tested. How do the authors imagine their method to extend or not extend in real-world domains with far more ambiguous linguistic commands?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KXxnG4zUAD", "reviewer": "Reviewer_uVPW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new way to perform robot evaluations for language instruction-based tasks. Specifically, it looks at language and scene perturbations, and explicitly consider the cost of making these evaluations. The goal is make evaluations of robot policies more accurate and cheaper.", "summary_of_recommendation": "Overall, I appreciate this paper’s effort in finding cheap proxies for performance of robot policies. While the scope is limited and specific, the proposed contrast sets would likely still be useful when a wider class of perturbations is considered.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- What are other types of scene and language perturbations that this evaluation framework can capture?\n- Do contrast sets enable more efficient evaluations of existing large pre-trained models like DROID and OXE/Octo?\n- Defining the cost for changes in the object positions is natural because the space is continuous. How should cost be measured for discrete perturbations like swapping a new object instance into the scene?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EIrURcJ26O", "reviewer": "Reviewer_DoyM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an approach to efficient real-world robot evaluation using contrast sets to perturb a pre-determined set of evaluation criteria (in terms of state and language input). The perturbed evaluations should be reflective of larger scale evaluations, while optimizing for minimum experimenter effort. Experiments are done on a simulated manipulation task and a real-world evaluation task.", "summary_of_recommendation": "Overall, the paper presents a good idea towards efficient on-robot evaluation. However, the presentation of the method and experiments needs further clarity. The experiments also need to be expanded to compare standard evaluation with many more evaluations than the contrast set method and a baseline that does a balanced standard evaluation with fewer evaluations than the contrast set method. With these improvements, the paper will be a good contribution to the robot learning community.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "In addition to the weaknesses listed above, I have the following questions for the authors.\n1. Why did you choose to perturb the initial state and the language? Are there other options? Would they be worth considering?\n2. Do you have a reference of a paper that evaluated policies on thousands of instructions? Although that is the goal of embodied foundation models, I am not sure that that has really been achieved yet. If not, I would just reframe these statements as - we are going towards this evaluation scheme.\n3. Is the mean in Fig. 3 over the unperturbed evaluation set? If so, why is it above the 'Original' bar plot?\n4. What is the number of standard evaluations that could be achieved with 1 m of cost (w.r.t. the discussion in lines 255-258)?\n5. How can the cost of the contrast set be less than the cost of the standard evaluation when the former is a superset of the latter?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a framework for real-world robot evaluation using contrast sets to perturb a pre-determined set of evaluation criteria and explicitly considers the cost of making these evaluations to optimize for minimum experimenter effort.

Strengths: The reviewers agree on the importance of the problem setup.

Weaknesses: The main criticism lies in the scalability of the proposed approach to wider data distributions and models and improving the clarity of the presentation.

Post response period: The authors have addressed some of reviewer's concerns. In particular, they have improved the overall clarity of the presentation. However, adding more comprehensive experimental results in more environments/datasets would help make the paper stronger. Given the author response, the updated ratings and their justifications, the AC recommends the paper to be accepted with a poster presentation.
Decision: Accept