RoCUS: Robot Controller Understanding via Sampling
Authors: Yilun Zhou
Serena Booth
Nadia Figueroa
Julie Shah
Venue: CoRL2021 Poster (2021)
Keywords: debugging and evaluation
algorithmic transparency

Abstract: As robots are deployed in complex situations, engineers and end users must develop a holistic understanding of their behaviors, capabilities, and limitations. Some behaviors are directly optimized by the objective function. They often include success rate, completion time or energy consumption. Other behaviors -- e.g., collision avoidance, trajectory smoothness or motion legibility -- are typically emergent but equally important for safe and trustworthy deployment. Designing an objective which optimizes every aspect of robot behavior is hard. In this paper, we advocate for systematic analysis of a wide array of behaviors for holistic understanding of robot controllers and, to this end, propose a framework, RoCUS, which uses Bayesian posterior sampling to find situations where the robot controller exhibits user-specified behaviors, such as highly jerky motions. We use RoCUS to analyze three controller classes (deep learning models, rapidly exploring random trees and dynamical system formulations) on two domains (2D navigation and a 7 degree-of-freedom arm reaching), and uncover insights to further our understanding of these controllers and ultimately improve their designs. 

RoCUS: Robot Controller Understanding via Sampling
Yilun Zhou
Serena Booth
Nadia Figueroa
Julie Shah
MIT CSAIL
(yilun, serenabooth, nadiafig, julie a shah)@csail mmitt.iiiiii......m.. edu
https: http:://yylluzzhougiiheeee
Abstract: As robots are deployed in complex situations, engineers and end users must develop a holistic understanding of their behaviors, capabilities, and limita- tions. Some behaviors are directly optimized by the objective function. They often include success rate, completion time energy consumption. Other behaviors e.g., collision avoidance, trajectory smoothness or motion legibility are typically emergent but equally important for safe and trustworthy deployment. Designing an objective which optimizes every aspect of robot behavior is hard. In this paper, we advocate for systematic analysis of a wide array of behaviors for holistic un- derstanding of robot controllers and, to this end, propose a framework, RoCUS, which uses Bayesian posterior sampling to find situations where the robot con- troller exhibits user-specified behaviors, such as highly jerky motions. We use RoCUS to analyze three controller classes (deep learning models, rapidly explor- ing random trees and dynamical system formulations) on two domains (2D nav- igation and a degree-of of-frredom arm reaching), and uncover insights to further our understanding of these controllers and ultimately improve their designs.
Keywords: Debugging and Evaluation, Algorithmic Transparency
1 Introduction
In 2018, after a confluence of failures, an autonomous vehicle (AV) struck and killed pedestrian for the first time. In the run-up to this fateful event, the responsible company had reportedly been trying to improve the AV "ride experience" by emphasizing non-critical behaviors-such as the smoothness of the ride [1]. This event reflects the long-standing challenge in robotics: designing an appropriate objective which considers both safety-critical and non-critical behaviors. When crafting an objective, it is virtually impossible to proactively account for all potential controller behaviors, and some priorities may even be in conflict with one another [2]. In practice, any given robot behaviors may be specified, unspecified, or even misspecified [3], so extensive testing and evaluation critical component of designing and assessing robot controllers especially those using black- box models such as deep neural networks.
A common testing procedure focuses finding extreme and edge cases of controller failure. For example, a tester might use this procedure to find that the AV swerves very badly when encounter- ing farm animal while traveling 60mph. Finding such extreme and edge cases is well-studied within both traditional software testing paradigms [4] and more recent adversarial perturbation test- ing methods [5]. However, we argue that an equally, if not more, important form of testing should focus on representative scenarios, which considers the likelihood of encountering these scenarios. For example, if this AV is going be deployed exclusively New York City, the above example is largely unhelpful: cars rarely travel at 60mph in the city, and are very unlikely to encounter farm an- imals. Instead, the tester may prefer to know that the car swerves -though not as substantively at lower speeds when a pedestrian steps toward it. Finding representative scenarios is often overlooked, but is especially useful for robotics. This is the focus of this paper.
Explicit mathematical analysis of robot controllers is implausible given the high dimensionality of the configuration space and the potential black-box representation of a learned controller. With access to an environment simulator, though, a straightforward testing approach is to roll out the robotic controller on various environments (c.g. road conditions under different weather and con- gestion, with or without farm animals pedestrians, etc.), and analyze those rollouts that exhibit
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

specified behavior- like excessive swerving. However, with too few environments, we risk miss- ing the condition(s) that triggers the target behavior most saliently. With too many environments, all the most salient rollouts would be close to the global maximum at the expense of diversity and coverage. For example, if farm animal causes the most swerving, followed by a pedestrian and a dangling tree branch, using too few environments may only find the pedestrian and the tree branch while using too many would result in an exclusive focus on the farm animal. Neither case helps the human develop correct mental model of the AV's behavior.
To address this, we introduce Robot Con- troller Understanding via Sampling (RoCUS), method enable systematic behavior inspec- tion. ROCUS finds scenarios that are both in- herently likely and elicit specified behaviors by formulating the problem as one of Bayesian posterior inference. Analyzing these scenarios and the resulting trajectories can help develop- ers better understand the robot behaviors, and allow them to iterate on algorithm development undesirable ones are revealed.
Figure Two use case demos of ROCUS: 2D
navigation (left) and 7DoF arm reaching (right).
We use ROCUS to analyze three controllers on two common robotics tasks (Fig. 1). For 2D navigation problem, we consider imitation learning (II.) [6], dynamical system (DS) [7], and rapidly- exploring random tree (RRT) [8]. For 7DoF arm reaching problem, we consider reinforcement learning (RL) [9]. as well the same DS and RRT controllers. For each problem and controller, we specify several behaviors and visualize representative scenarios and trajectories that elicit those behaviors. Through this analysis, we uncover insights that would be hard to derive analytically and thus complement our mathematical understanding of the controllers. Moreover, we include a case study on how to improve a controller based on new insights from RoCUS. As such, RoCUS is a step towards the broader goal of building more accurate human mental models and enabling holistic evaluation of robot behaviors.
2 Related Work
Our work lies at the intersection of efforts understand complex model behaviors and those to benchmark robot performance. Methods to understand, interpret, and explain model behaviors are now commonplace in the machine learning community. Mitchell et al. [10] introduced Model Cards, a model analysis mechanism which breaks down model performance for data subsets. In natural language processing, Ribeiro et al. [11] introduced a checklist for holistic evaluation of model capa- bilities and test case generation. Booth et al. [12] introduced BAYES- TREX, a Bayesian inference framework for sampling specified classifier behaviors. In robotics, Fan et al. [13] introduced a ver- ification framework for assessing machine behavior by sampling parameter spaces to find temporal behaviors. Other efforts aim to summarize robot policies, trading off factors like brevity, diversity and completeness [14, 15]. All of these works have shared underlying theme: treating the black box as immutable and performing downstream analyses of machine behavior [16]. ROCUS shares this theme and, similar to MAYES-TREY [12], searches for instances which exhibit target behaviors to inform accurate human mental models.
While the need for benchmarking robot performance is often expressed [17, 18, 19], these efforts usually operate distributions of trajectories randomly selected trajectories, and the accompa- nying metrics are typically task-completion based without consideration of implicit performance factors. Anderson et al. [20] put forth a recommendation of using success weighted by path length
for navigation tasks-a fask-completion metric. Cohen et al. [21] and Moll et al. [22] introduced suites of metrics for comparing motion planning approaches, and Lagriffoul et al. [23] presented a set of task and motion planning scenarios and metrics. Again, all of these proposed metrics are based solely on task completion. Lemme al. [24] proposed a set of performance measures for reaching tasks, which are either cusk-completion based or require costly human motion ground truth. Our contribution is distinct two ways. First, we propose to sample specific trajectories which communicate controller behaviors instead of reporting metrics averaged over distributions of trajectories. Second, we introduce metrics which draw on these prior works while also including essential alternative and typically emergent quality factors, like motion jerkiness and legibility [25].

---

3 ROCUS
At high level, RoCUS helps users understand robotic controllers via representative scenarios that exhibit vari- ous specified behaviors. It solves this by directly incorpo- rating the distribution of scenarios, formally called tasks, into a Bayesian inference framework as shown in Fig. 2. A robotic problem is represented by distribution T()) of individual tasks t. For example, a navigation problem may have xt)) representing the distribution over target locations and obstacle configurations. Given a specific task t, the controller under study induces a distribution p(r(t) of possible trajectories If both the controller and the transition dynamics deterministic, p(T(t) reduces to a 8- 5-function at the induced trajectory т. Stochasticity in either the controller (e.g., RRT) or the dynamics (e.g., uncertain outcome from action) can result in be- ing random. Finally, a behavior function b(T,t) computes the behavior value of the trajectory- for example, the motion jerkiness. Some behaviors only depend the trajectory and not the task, but we use b(T, 1) for consistency. Sec. 4 presents a list of behaviors.
The graphical model for the inference problem of finding tasks and trajectories which exhibit specific be- haviors The dashed box indicates the relaxed formulation (Eq. 2).
Figure 2:
The discussion
behavior in Sec. 1 is informal and implicitly combines two related but different concepts. The first concept is the behavior function b(T, t) discussed above. The second is the spec- ified target: for the swerving example, Thus, the target value can be thought of as +00. This inference problem uses the maximal mode of ROCUS. In other cases, target. For example, we want to find road conditions that lead to a daily commute time of an hour, where the behavior is the travel time. This inference problem uses the matching mode. Since match- ing mode is conceptually simpler, we present it first, followed by maximal mode. The sampling procedure is the same for both modes and presented last in Alg.
are particularly interested in maximal behavior values.
also interested in tasks and trajectories whose behaviors matches
3.1 Matching Mode
The exact objective is to find tasks and trajectories that exhibit user-specified behaviors b*:
1 ~ p(4,r|b= b²) x ((())))()))))((())()
In most cases this posterior does not admit direct sampling, and an envelope distribution is not available for rejection sampling. Markov-Chain Monte-Carlo (MCMC) sampling does not work either: since the posterior is only non zero on a very small or even measure-zero set, a Metropolis- Hastings (MH) sampler [26] can get stuck in the zero-density region. Similar to the formulation [12), we relax it using a normal distribution formulation as shown in Fig. 2:
blb (b, f p(t, r/b b*) p(b b" [t, ())())))=(t) This relaxed posterior is non-zero everywhere xtt) is non-zero and provides useful guidance to an MH sampler. While a is a hyper-parameter in BAYES-TRES [12), we instead choose a such that
(2)
0'+
| p(b) db = a, with p(b) = J. (()))) (6),)) dr dt
(3)
being the marginal distribution of b(T, 1), which can be estimated by trajectory roll-outs. This formu- lation has two desirable properties. First, it scale-invariant with respect to b(T, (), e.g. measured under different units like meters vs. centimeters. Second, the hyper-parameter o E [0, 1] has the intuitive interpretation of the approximate "volume" of posterior samples f, b 6* under the marginal p(t, r) = p(T[t))((t), a notion of their representativennss. Details are derived App.
3.2 Maximal Mode
In this mode, RoCUS finds trajectories that lead maximal behavior values: by be used for finding minimal behavior values by negating the behavior. The posterior formulation is:
too. It can also
E
bo
B
Ite N~
t, p(t, - 1),
(4)
b
where bbb and bbb are the mean and variance of the marginal p(b). a is chosen such that
Jh p(3) d3 = a,
(5)
3

---

where p(3) is the marginal distribution similar to Eq. 3. If p(b) is normal, p(3) is logit-normat This formulation again
and has the same "volume" interpretation for (App. A).
3.3 Posterior Sampling
Algorithm 1: MH Sampling Procedure
The posterior sampling mechanism depends on the stochasticity of the controller and dynamics.
Input: "Posterior volume" a, number of samples N, optional burn-in NB and thinning period Nr.
Deterministic Controller & Dynamics: When both the controller and the dynamics are de- terministic, so is T|t, denoted as (t). Eq.
samples []:
Get a from a by Eq. (matching) or (maximal);
2 reduces to t ~ p(t[b = b*) p(8 = 6" [t, (t)) and similarly for Eq.
Randomly initialize t;
for i- L....,N N do
Alg. presents the MH sampling procedure. First, a is computed from (Line 2). Then we start with an initial task (Line 3). For each of the N iterations, we propose a new task fnew according to a transition kernel and compute the forward and reverse transition probabilities
<now: Ptors Prev = propose(t)
Get p from by Eq. 2 (match) or Eq. 4 (max)
Get Pnew from fnew by Eq. 2 or Eq. 4; a +- (Pnew -ree Plor):
Sample
[1 1];
Prev (Line 5). We evaluate the posteriors under and fnew (Line and 7) and calculate the acceptance probability using the MH de- tailed balance principle (Line 8). Finally, we accept reject accordingly (Line 9 - 11). Note that if the proposal is rejected, the current is left unchanged and appended to the samples. We can discard the first NB samples as burn- in, and/or thin the samples by a factor of Nr to reduce auto-correlation.
10
if u < a then t fnew:
12
Append & to samples;
Optionally, discard the first Ng burn-in samples and thin the samples by only keeping every Nr samples; 14 return samples
When the controller and p(r]t) are stochastic, the controller can usually be implemented by sampling a random variable u (independent from f), and then pro- ducing the action based on the realization of u. as shown in Fig. 3. For instance, a Normal stochastic policy (s) (N((s)) (s))2) can
Stochastic Controller:
i
The same graphical model as in Fig. 2, but with the addition of stochasticity u in the troller such that rlt, u is now deterministic.
Figure 3:
be implemented by first sampling ~ (O(0 1)
and then computing (ss) p(s) s) + u (s)).
In this case, we sample in the combined (t, r-sppcee with Eq. being p(t. r = b*) x p = 8" It, r(e, ()))(u)=(t), where overload r(t, to refer to the deterministic trajectory given the task & and controller randomness u. It is crucial that for any u, we can evaluate p(u). Concretely, modifying Alg. 1, Unew is proposed alongside with nnow (Line 5), the detailed balancing factor (Line 8) is multiplied by PP.rrw//PP for, and know; Unew are accepted or rejected together (Line 10- 10-17 12).
Stochastic Dynamies: Using the same logic, RoCUS can also accommodate dynamics stochastic- ity, as long as it can be captured in a random variable u and p(u) can be evaluated. We leave the details to App. B and use deterministic dynamics in our experiments.
3.4 The Bayesian Posterior Sampling Interpretation
ROCUS uses Bayesian sampling concepts of prior, likelihood, and posterior quite liberally. Specif- ically, the task distribution is defined as the prior, and thus the notion of a task being likely in the deployment context refers to high probability under the prior. Likelihood refers to the behavior saliency: how much the exhibited behavior matches the behavior specification. The act of posterior sampling then finds tasks that strike a balance between these two objectives.
The choice of explicitly modeling the task distribution is intentional, as it is not unlikely that the deployment environment will be different than the development environment. Such domain mis- match may cause catastrophic failures, especially for learned controllers whose extrapolation be- haviors are typically undefined. With a suitable task distribution, RoCUS allows more failures to surface during this testing procedure.

---

4 Behavior Taxonomy
Robot behaviors broadly belong to one of two classes: intentional and emergent. Intentional be- haviors are those that the controller explicitly optimize with objective functions. For example, the controller for a reaching task likely optimizes to move the end-effector to the target, by setting the larget as an attractor in DS, using a target-reaching objective configuration in RRT, or rewarding proximity in RL. Thus, the final distance between the end-effector and the target an intentional behavior for all three controllers. By contrast, emergent behaviors not explicitly specified in the objective. For the same reaching problem, an RL policy with reward based solely on distance may exhibit smooth trajectories for some target locations and jerky ones for others. Such behaviors may emerge due to robot kinematic structure, training stochasticity, or model inductive bias.
line integral f, ((x) ds of version ||-|| J,, ((x) ds, ds, where ds is the infinites- imal segment on 7 at x and ||T|| is the trajectory length. x and T can be in either joint space or task space. We introduce six behaviors: length, time derivatives (velocity, acceleration and jerk), straight- line deviation, obstacle clearance, near-obstacle velocity and motion legibility, whose mathematical expressions are in App. C. In addition, custom behaviors also be used with RoCUS.
For trajectory many behavior metrics b(T, 1) can be expressed
scalar field V (xx along its dength-normalin
5 ROCUS Use Case Demos
In this section, we demonstrate how RoCUS can find "hidden" properties of various controllers for two common tasks, navigation and reaching. We also uncover a suboptimal controller design due to bad syper-paramette choices, which is improved based on RoCUS insights.
5.1 Controller Algorithms
We consider four classes of robot controllers. The imitation learning (IL) controller uses ex- pert demonstrations learn neural network policy which maps observations deterministic ac- tions. The reinforcement learning (RL) controller implements proximal policy gradient (PPO) [27]. While a mean and a variance is used to parameterize PPO policy during training. the policy deterministically outputs the mean action during evaluation. The dynamical system (DS) controller modulates the linear controller u(x) = x x, for the task-space target x", into uM (x) = M u(x) using the modulation matrix M derived from obstacle configuration, as proposed by Huber et al. [7]. We give a self-contained review in App. D. The appdll--xploriing
random tree (RRT) controller finds configuration-spae space trajectory via RRT and then controls the robot through descretized seg- ments. Notably, RRT is stochastic, and we discuss the use of controller stochasticity u (c.f. Fig. 3) in App. E. The MCMC sampling uses Gaussian drift kernel, detailed in App. F.
5.2 2D Navigation Task Experiments
Tree Path
Original Smoothed
Setup In rectangular arena with irregu- larly shaped obstacles, point mass robot needs to move from the lower left to the upper right corner (Fig. left). App. details the obstacle generation and robot simulation procedures and contains more environment visualizations.
ART
Smoathing and Lidar
DS Modulation
Figure 4: RRT, IL and DS controllers on 2D navi- gation domain. Left: the RRT controller tree. Mid- dle: smoothed RRT trajectory and lidar sensor (orange lines) for IL controller training. Right: the modulation by the DS controller.
We consider three controllers for this en- vironment: an RRT planner, a deep learn- ing IL policy, and a DS (Fig, 4). The RRT planner implements Alg. 2 and discretizes the path small segments as control signals each time step. The controller uses smoothed RRT trajectories as expert demonstrations, and learns to predict heading angle from its current position and lidar readings. The DS controller finds an interior reference point for each obstacle, and converts each obstacle in the environment to be star-shaped. T-functions are then defined for these obstacles and used to compute the modulation matrix M. App. H contains additional implementation details.
Straight-Line Deviation In most cases, the robot cannot navigate directly to the target in straight line. Thus, the collision-avoidance behavior is crucial aspect for navigation robots. To understand it, we sample obstacles that lead to trajectories minimally deviating from the straight line path. Since the deviation is always non non-negatinn -egative,, we use the matching mode in Eq. with target b* 0.
5

---

In Fig. 5, the top row plots posterior trajecto- ries orange, with prior trajectories in blue. The bottom row plots the obstacle distributions compared to the prior, with red regions being more likely to be occupied by obstacles and blue ones less likely to be obstructed.
For DS and RRT, the posterior trajectories and obstacle configurations are mostly symmetric with respect to the straight-line connection, as expected since both methods are formulated symmetrically with respect to the and y- coordinates. The obstacle distribution under RRT is also expected, since it seeks straight- line connections whenever possible and thus favor "diagonal corridor" with obstacles either side. For DS, however, obstacles are slightly more likely to exist the two ends of the above-mentioned corridor. This behavior is an artifact of the DS tail effect, which drags the robot around the obstacle (details in App. D). By taking advantage of anchor- like obstacles at the ends of the corridor, the modulation can reliably minimize the straight-line deviation.
Dynamical System
Ra Rapidy-Exxplaiing Random Tree
Imitation Learning
Posterior trajectories in orange vs. prior in blue for minimal straight-line devia- tion behavior for three controllers. Bottom: Pos- terior obstacle distribution relative to the prior. Higher obstacle density regions are painted in red and lower ones in blue.
Figure 5: Top:
By comparison, the IL controller saliently exhibits trajectory asymmetry: it mostly takes paths on the left. is possible that the asymmetry due to "unlucky" samples by the MH sampler, but many independent restarts all confirm its presence, indicating that the asymmetry is inherent in the learned model. Since the neural network architecture symmetrie, we conclude that the stochasticity in the dataset generation and training procedure (e.g. initialization) leads to such imbalanced behav- iors. Furthermore, the obstacle map suggests that obstacles are distributed very close to the robot path. Why does the robot seem to drive into obstacles? The answer lies in dataset generation: the smoothing procedure (Fig. 4 middle) results in most demonstrated paths navigating tightly around obstacles, and it is thus expected that the learned IL controller displays the same behavior.
Takeaways ROCUS reveals two unexpected phenomena. First, IL trajectories are highly asym- metric toward the left of the obstacle due to dataset and/or training imbalance. Second, both DS and IL models exhibit certain cobstacle-seeking behaviors, the former due to the "tail-effect" and the latter due the dataset generation process. In both cases, such behavior may be undesirable in deployment due to possibly imprecise actuation, and the controller design may need to be modified. Additional studies on legibility and obstacle clearance behaviors are presented in App. I.
5.3 7DoF Arm Reaching Task Experiments
Setup A 7DoF Franka Panda is mounted the side of table with a T-shaped divider (Fig. right). Starting from the same initial configuration on top of the table, needs to reach a random location either side under the divider. We simulate this task in PyBullet [28]. We consider three controllers: an RRT planner, a deep RL PPO agent, and DS formulation.
RRT again implements Algorithm 2, but uses inverse kinematics (IK) first find the joint config- uration corresponding to the target location. The RL controller is a multi-layer perceptron (MLP) network trained using the PPO algorithm. The DS model outputs the end-effector trajectory in the task space, which is converted to joint space via IK, with SVM-learned obstacle definitions. App. J contains additional implementation details for each method. Overall, RRT and RL are quite suc- cessful in reaching the target while the DS is not due to the bulky robot structure, close proximity to the divider, and the task-space only modulation.
End-Effecte Movement We find configurations that minimize the total travel distance of the end- effector for RRT and RL (DS omitted due high failure rate). Fig. 6 (left two) shows the posterior target locations and trajectories. Notably, unlike RL, RRT trajectories are highly asymmetric, since there are straight-line connections the configuration space from the initial pose to some target regions on the left, while every right-side goal requires at least an intermediate node.
DS Improvement with ROCUS Our initial DS implementation frequently fails to reach the target. This is understandable, as the DS convergence guarantee [7] is only valid in task space, in which

---

RRT Min EE Movement
RL Min EE Movement
Original DS Final Distance
Improved Final Distance
Figure 6: Left: Minimal end-effector movement samples for RRT and RL. Right: Posterior samples for minimal distance from end-effector to target for the original and improved DS controllers. Top: posterior targets locations, with tabletop divider in green and target region in orange. Bottom: posterior trajectories in red, prior trajectories in blue. Robot is mounted on the near long edge.
the modulation is defined. When the full-arm motion is solved via IK, is possible that some body parts may collide and get stuck because of the table divider. To understand the DS behaviors, ROCUS to sample target locations that result in minimal final distance from the end-effector to the target (i.c., most successful executions, Fig. center-right). Similar the RRT case, the samples show strong lateral asymmetry, with all posterior target locations on the left, due to the same cause of asymmetric kinematic structure. The result points to a clear path improve the DS controller such that it can succeed with right-side targets: increase the collision clearance of the divider so that the end-effector navigates farther away from the divider, thus also bringing the whole arm be farther away. As detailed in App. K, this modification greatly improves the controller performance as confirmed by the new symmetry Fig. (rightmost). addition, since the issue with DS controller mainly lies in obstacle avoidance in joint-space (29, 30, 31, 32] could used and we leave them future directions.
the body of the robot, additional techniques
Takeaway The set of studies reveal an important implication of the robot's kinematic structure: the left side is much less "congested" with obstacles than the right side in the configuration space. While the RL controller is able learn efficient policies for both sides, the design of certain controllers may need to explicitly consider such factors. App. K includes an additional study on legibility.
5.4 Quantitative Summary
We studied other additional behaviors on both tasks, and Tab. 1 summarizes prior posterior mean behavior values and shows that RoCUS consistently finds samples salient in the target behavior.
Domain
Behavior
Target
Prior (DS)
Post. (DS)
Prior (IL/RL)
Post. (IL/RL)
Prior (RRT)
Post. (RRT)
Avg. Jerk 0
1.84e-3
1,46e-3
6.95e-4
3.19
4,24e-4
2. 2.79e-4
Straight
0
0.256
0.084
0.378
0.301
0.470
0.162
2D Nav
Obstacle
Legibility
min
0.309
0.819
0.229
0.650
0.262
0.877
0.218
0.784
0.798
0.312
0.669
0.241
Obstacle
max
0.309
0.611
0.262
0.387
0.312
0.442
Straight
0
0.980
0.913
0.858
0.762
1.223
0.897
Arm
EG Dist
0
0.934
0.623
0.958
0.691
3.741
1.192
Table 1: Quantitative results on additional tasks for two domains.
6 MCMC Sampling Evaluation
Behavior Value
After confirming that RoCUS can indeed uncover significant and actionable controller insights, we evaluate the sampling procedure itself, using tasks described above examples.
0.1
Mixing Property A potential downside of MCMC sam- pler the slow mixing time, which causes the chain to take a long time to converge from initialization and causes consec- utive samples to be highly correlated. Does this phenomenon happen for sampling? Fig. 7 plots the behavior along the
5000
10000
MCMC Iteration
Figure 7: 2D navigation DS min straight-line deviation samples.

---

MCMC iterations for the DS minimal straight-line deviation behavior, showing that the chain mixes well quite fast (additional ones in Fig. 10 of App. F). Thus, a modest amount of samples, such as several thousand, is typically sufficient to model the target posterior distribution well.
Baseline: Top-k Selection To the best of our knowledge, RoCUS is the first work that applies the ciissparency-b-cxxmppl formulation [12] to robotic tasks, and we are not aware of existing meth- ods for the same purpose. Notably, adversarial perturbation algorithms [5] are not feasible, since stepping in simulator (or real world) is not typically differentiable. Sec. discusses a straightforward alternative that runs the controller on N different scenarios and pick the top-k with respect to the target behavior. We demonstrate its shortcomings on the minimal straight-line deviation behavior for the 2D navigation DS controller (RoCUS samples shown in Fig. 5 left).
Fig. (left) shows the trajectories of different values of for the same fixed N, and vice versa. While bigger N/k ratio leads to more salient behaviors in the top-k samples, these examples become more concentrated around the global maximum and less diverse, making this approach especially myopic. Further, it is not easy find the optimal N to trade off between diversity and saliency of the top-k samples, By contrast, ROCUS offers the intuitive haper-parameter. Fig. 8 (middle) shows that a smaller N fails highlight the "corridor" pattern while larger N makes completely open and misses the "tail-effect anchors" at the two ends.
In addition, the hard cut-off at the k-th salient behavior threshold has two undesirable implications: first, every trajectory more salient than the threshold is kept but is given equal importance; second, trajectory even slightly under the threshold is strictly discarded. By comparison, RoCUS gives more importance to more salient samples in a progressive manner, as shown in Fig. 8 right.
Finally, top-k selection is very computationally inefficient. It discards all of the unselected N k samples, while RoCUS is much more efficient in that all samples after the burn-in up to the thinning factor can be kept since the posterior concentrated on the salient behavior is directly sampled.
Different k. same N-10000
Same k-50, different N
k=50, N=100
k- 50, N- 1-10000
Behavior value p.d.f.
N::50
20
Top-k
N-1000
15
RUCUS
No 10000
10
k-3000
K-1000
k-10
Figure 8: Top-k selection baseline. Left two: trajectory distribution; middle two: obstacle distribu- tion; right one: probability density function of behavior values.
Discussion and Future Work
RoCUS enables humans to build better mental models of robot controllers. Compared to existing evaluations task-completion metrics for hand-designed tasks, ROCUS generates tasks and tra- jectories that highlight any given behavior in a principled way. We used it to uncover non-obvious insights in two domains and help with debugging and improving a controller.
While RoCUS is mainly tool to analyze robot controllers simulation as part of comprehensive testing before deployment, it can help understanding (anomalous) real world behaviors as well. When an anomaly is observed, RoCUS can find more samples with the anomaly for developers to identify patterns of systematic failures. Furthermore, RoCUS is not inherently limited to simulation: only requires trajectory roll-out on specific tasks. For the arm reaching task, this is easy in the real world. For autonomous driving, "recreating" a traffic condition that involves other vehicles may be hard. However, a key feature of RoCUS is the decoupling of the task and the controller algorithm, which allows testing on simpler task variants (e.g. with props instead of real cars).
There are multiple directions for future work, including evaluation of model updates [33] by defining behavior functions on two controllers, better understanding the samples with explainable artificial intelligence (XAI) methods, and an appropriate interface to facilitate the two-way communication between ROCUS and end-users, as discussed in detail in App. L.
Overall, ROCUS is framework for systematic discovery and inspection of robotic controller be- haviors. We hope that the demonstrated utility of RoCUS sparks further efforts towards the devel- opment of other tools for more holistic understanding of robot controllers.
8

---

Acknowledgement
This research supported by the National Science Foundation (NSF) under the grant IIS-1830282. We thank the reviewers for their reviews, which are available at https://opennee https://ppenreiiee..ntt Srtriddd5P-bbWWiRS.
References
[1] Bort. Inside Uber before its self-driving car killed pedestrian: Sources de- scribe infighting, "perverse" incentives, and questionable decisions, 2018. URL and
itt.rruuueettiaabbeeeeeiiiiiisss delii-iecssiiss- http://iws..dr...oooooooeeeedeee - www---ee-----drriiinneeee-- wnttriinnntit-befrreeee o iir-its-ccrss tee-pedessrinn-211185
ubside-
[2] A. Rasouli and J. K. Tsotsos. Autonomous vehicles that interact with pedestrians: survey of theory and practice. IEEE Transactions on Intelligent Trunsportation Systems (T-ITS), 21)3): 900-918, 2019.
[3] A. Bobu, A. Bajesy, J. F. Fisac, S. Deglurkar, and A. D. Dragan. Quantifying hypothesis space misspecification in learning from human-robot demonstrations and physical corrections. IEEE Transactions Robotics (T-RO), 36(3))835-8544 2020.
[4] G. J. Myers, T. Badgett, T. M. Thomas, and C. Sandler. The Art of Software Testing, volume 2. Wiley Online Library, 2004.
[5] I. J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples. In International Conference Learning Representation (ICLR), 2015.
[6] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems (RAS), 57(5):469-483 2009.
[7] L. Huber, A. Billard, and J.-J. Slotine. Avoidance of convex and concave obstacles with con- vergence ensured through contraction. IEEE Robotics and Automation Letters (RA-L), 4(2): 1662,1469.00 2019.
[8] S. M. LaValle. Planning Algorithms. Cambridge University Press, 2006.
[9] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[10] M. Mitchell, S. Wu, A. Zaldivar, Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, D. Raji, and T. Gebru. Model cards for model reporting. In Conference Fairness, Accountability; and Transparency (FAT*), ), pages 220-229, 2019.
[11] M. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4902-4912, 2020.
[12] Booth, Y. Zhou, A. Shah, and J. Shah. Bayes-trex: a bayesian sampling approach to model transparency by example. In AAAI Conference Artificial Intelligence (AAAI), volume 35, pages 11423-11432, 2021.
[13] C. Fan, X. Qin, and Deshmukh. Parameter searching and partition with probabilistic cover- age guarantees. arXiv preprini arXiv:2004.00027,, 2020.
[14] B. Hayes and J. A. Shah. Improving robot controller transparency through autonomous pol- icy explanation. ACM/IEE International Conference on Human- Robot Interaction (HRI), pages 303-312. IEEE, 2017.
[15] I. Lage, D. Lifschitz, F. Doshi-Velez, and O. Amir. Exploring computational user models for agent policy summarization. In International Joint Conference on Artificial Intelligence (IJCAI), pages 1401-1407, 2019.
[16] I. Rahwan, M. Cebrian, N. Obradovich, J. Bongard, J.-F. Bonnefon, C. Breazeal, J. W. Cran- dall, A. Christakis, D. Couzin, M. O. Jackson, et al. Machine behaviour. Nature, 568 (7753):477-488 486, 2019.

---

[17] J. Mahler, R. Platt, A. Rodriguez, M. Ciocarlie, A. Dollar, R. Detry, M. A. Roa, H. Yanco, A. Norton, J. Falco, et al. Guest editorial open discussion of robot grasping benchmarks, protocols, and metrics. IEFE Transactions Automation Science and Engineering (T-ASE), 14(4):1440-1442, 2018.
[18] A. Murali, T. Chen, K. V. Alwala, D. Gandhi, L. Pinto, S. Gupta, and A. Gupta. Py- Robot: An open-source robotics framework for research and benchmarking. arXiv preprint 100.00/900 2019.
[19] S. James. Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters (RA-L). 5(2):301933026, 2020.
[20] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018.
[21] B. Cohen, I. A. Şucan, and S. Chitta. A generic infrastructure for benchmarking motion plan- ners. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 589-595, IEEE, 2012.
[22] M. Moll, I. A. Sucan, and L. E. Kavraki, Benchmarking motion planning algorithms: An ex- tensible infrastructure for analysis and visualization. IEEE Robotics & Automation Magazine (RA-M), (233):96-10,, 2015.
[23] F. Lagriffoul. N. T. Dantam, C. Garrett, A. Akbari, S. Srivastava, and L. E. Kavraki. Platform- independent benchmarks for task and motion planning. IEEE Robotics and Automation Letters (RA-L), 3(4):3765-377) 3772, 2018.
[24] A. Lemme, Y. Meirovitch, M. Khansari-Zadeh, T. Flash, A. Billard, and J. J. Steil. Open- source benchmarking for learned reaching motion generation in robotics. Paladyn, Journal of Behavioral Robotics, 6(1), 2015.
[25] A. D. Dragan, K. C. Lee, and S. S. Srinivasa. Legibility and predictability of robot motion. In ACM/EEEE International Conference on Human-Robot Interaction (HRR), pages 301-308. IEEE, 2013.
[26] W. K. Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika, 57(1), 1970.
[27] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 177..03777.0647 2017.
[28] E. Coumans and Y. Bai. PyBullet, a python module for physics simulation for games, robotics and machine learning. http://pybbulettorgg 2016-2021.
[29] O. Khatib. Real-time obstacle avoidance for manipulators and mobile robots. In IEEE Inter- national Conference on Robotics and Automation (ICRA), volume 2, pages 500-505. IEEE, 1985.
[30] N. D. Ratliff, J. Issac, D. Kappler, S. Birchfield, and D. Fox. Ricmannian motion policies. arXiv preprint 1rXiv:1801.02854 2018
[31] S. S. Mirrazavi Salchian, N. Figueroa, and A. Billard. A unified framework for coordinated multi-arm motion planning. The International Journal of Robotics Research (IJRR), 37(0)): 1205-1232, 2018.
[32] J. Urain, P. Liu, A. Li, C. D'Ermoo, and J. Peters. Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning. In Roboties: Science and Systems (RSS), Virtual, 2021.
[33] G. Bansal, B. Nushi, E. Kamar, D. S. Weld, W. S. Lasecki, and E. Horvitz. Updates in human- AI teams: Understanding and addressing the performancecompatibliig tradeoff. In AAAI Conference on Artificial Intelligence (AAAI), volume 33, pages 2429-2437 2019.
10

---

[34] S. Brooks, A. Gelman, G. Jones, and X.-L. Meng. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
[35] M. Khansari-Zadeh and A. Billard. A dynamical system approach to realtime obstacle avoidance. Autonomous Robots (AuRo), 32(4):433-455 2012.
[36] Y. Zhou and K. Hauser. 6DOF grasp planning by optimizing deep learning scoring function. In RSS Workshop on Revisiting Contact - Turning a Problem into a Solution, 2017.
[37] H. Xu, Y. Gao, F. Yu, and T. Darrell. End-to-end learning of driving models from large-scale video datasets. IEEE conference on computer vision and pattern recognition (CVPR), pages 2174-2182, 2017.
[38] K. Hauser. Robust contact generation for robot simulation with unstructured meshes. In 16th International Symposium on Robotics Research (ISRR), volume 14, page 357. Springer, 2016.
[39] Karaman and E. Frazzoli. Sampling-based algorithms for optimal motion planning. The International Journal of Robotics Research (IJRR), 30(7):846-33) 894, 2011.
[40] K. Hauser and Y. Zhou. Asymptotically optimal planning by feasible kinodynamic planning state-cost space. IEEE Transactions on Robotics (T-RO), 32(6):1431-1443 2016.
[41] S. Greydanus, A. Koul, J. Dodge, and A. Fern. Visualizing and understanding Atari agents, In International Conference on Machine Learning (ICML), pages 1792-1801. PMLR, 2018.
[42] Zahavy, N. Ben-Zrihem, and S. Mannor. Graying the black box: Understanding DQNs. In International Conference on Machine Learning (ICML), pages 1899-1908. PMLR, 2016.
[43] A. Atrey, K. Clary, and D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. In International Conference on Learning Rep- resentations (ICLR), 2019.
[44] Zhou, S. Booth, M. T. Ribeiro, and J. Shah. Do feature attribution methods correctly attribute features? arXiv preprint arXiv:2104. 14403, 2021.


Reviews:
[{"review_id": "gem6uovdrZ", "reviewer": "Reviewer_h5ZA", "summary": "In this article, the authors propose a Bayesian framework that aims to find situation in robot planning tasks that lead to trajectories which exhibit specified behaviors. For this purpose, the task distribution is defined as the prior, and the behavior saliency is the likelihood. Then, the samples of the posterior are tasks that balance these two objectives. Two simulations demonstrate the benefits of the proposed approach. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tInteresting and time-relevant topic as it allows to study the behavior of “black-box” controller in robotics\n-\tSmart idea using the Bayesian framework\n- Simulation with complex examples\n-\tPaper is well written and structured\n\nWeakness\n-\tSome minor mistakes and missing definitions\n-  The baseline of the simulation seems to be too simple\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "One of the most interesting papers I have reviewed this year. The proposed approach is highly relevant as more and more \"black-box\" controller are used in robotics. However, the results should be compared to other more complex metrics.\n\nUpdate: Based on the authors' response, I updated my recommendation to \"strong accept\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- (2) What is No? I guess it's the normal distribution but the notation is uncommon and never introduced.\n- Alg.1 : \"numner\" -> \"number\"\n- Alg.1 : How does the choice of $N_B$ and $N_t$ affect the result?"}}, {"review_id": "gUSkJ0E0sLp", "reviewer": "Reviewer_U7Po", "summary": "This work aims at providing a systematic way to debug controllers and analyze behavior saliency, in order to avoid dangerous or unexpected situations. There are various approaches to do so, as direct testing of extreme and edge cases or providing a mathematical analysis of the controller. In most cases, this is either cumbersome or extremely difficult. The focus of the paper is to establish a framework to evaluate controllers and summarize robot performance with the specific purpose of informing accurate human models. One way of testing for controller failure is to roll it out on many different scenarios, but this is a notably difficult problem, considering that it involves sampling from a posterior which is predominated by zero-density regions (salient behaviors/edge cases). They present RoCUS, a method that enables understanding of robot controllers via posterior sampling of representative behaviors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written apart from some typos. They provide a solution to the problem of posterior sampling by relaxing the posterior and reformulate it such that one only has to tune an intuitive hyperparameter (alpha) which represents the volume of the samples. Like this, one can use methods like Metropolis-Hastings, as they do, to sample from the posterior. They also propose different alternative metrics for behavior understanding, that do not purely focus on task completion. \nOne of the weaknesses of the paper was a lack of a baseline comparison (with any of the methods cited in the related work for example). It could be nice to see how it improves over a less sophisticated form of sampling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper provides a complementary approach to direct testing of extreme cases, which is very useful when the controller is deployed in situations that could hurt humans. The two presented use cases (navigation and manipulation) provided interesting insights which could be further showcased and assessed in other more complicated environments, or better, more complicated tasks, in the lack of real robotic experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- Line 153: what do you mean when b tends to + or - infinity?\n- Line 312: what do you mean with b=0?\n- How does RoCUS compare to other task completion-based metrics?\n- While describing figure 2 you mention that for a deterministic system and controller, the probability of a given trajectory is a delta. It is not clear to me if you are modeling also the initial condition of the dynamical system, together with the trajectory. \n- Would be interesting to see more use cases of matching vs maximal mode.\n- The premises of the paper are very alluring and it would be interesting to see RoCUS applied to more environments/tasks."}}, {"review_id": "6ol3CrmP4M9", "reviewer": "Reviewer_UK3k", "summary": "The paper introduces a Bayesian model for policy evaluation. They introduce in a principled way an approach to sample tasks and trajectories in which a certain desired metric, b, is maximized. To do so, the paper proposes to apply Metropolis Hasting over an approximated posterior distribution.\n\nThe paper proposes a set of behavioural metrics (trajectory length, time derivatives, straight line deviation, obstacle clearance and motion legibility) and evaluate the performance for a set of policies (Imitation Learning, Motion Planning, Reinforcement learning and Dynamic Systems) to evaluate the proposed policy evaluation method. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Formulating policy evaluation for robust interaction with he environment as a bayesian inference problem is interesting as it provides well-formulated problem that allow us to get deeper insights and let us compare the different methods for policy evaluation.\n\nWeakness:\n1. The main weakness of the paper is the lack of qualitative metrics. The paper proposes an evaluation method, but only evaluates the performance of a set of policies with quantitative measures. \n2. RRT algorithm is well known and it would be enough to re-direct the reader to the original paper, rather than putting the algorithm itself in the paper\n3. When applied the Dynamics-based approach, they first generate a task space trajectory and then map the trajectory to the configuration space by Inverse Kinematics. Instead, the performance of the DS based method would improve if it considers a multi-objective reactive motion generator in which obstacle avoidance elements can integrated [1,2,3] with the goto movements.\n4. The paper lacks any information of the required amount of samples (from both trajectories and tasks) for proper evaluation.\n\n\n[1] Khatib, Oussama, \"Real-Time Obstacle Avoidance for Manipulators and Mobile Robots\", 1986\n\n[2] Ratliff, Nathan, et al., \"Riemannian Motion Policies\", 2018\n\n[3] Urain, Julen et al., \"Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning\", 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I suggest a weak rejection. The paper proposes a policy evaluation framework but it did not exploit it enough. The paper requires more metrics, more comparison, more policies and qualititative metrics for a better methods comparison.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "The authours might consider adding adding qualititative metrics to improve the policy evaluation.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "1Z-k8I4VRsk", "reviewer": "Reviewer_YUw7", "summary": "Complex robotic controllers typically demonstrate both intended and emergent behavior. Whilst intended behavior is optimized for and measured (e.g. navigate to x without collisions, grasp object y), often complex controllers exhibit emergent behaviors (e.g. jerkiness, object clearance, path length) that are also important to understand and measure. This work presents RoCUS, a method for understanding emergent controller behavior using a Bayesian sampling-based method. Unlike other controller testing techniques, RoCUS focuses on sampling representative trajectories instead of extreme or edge case trajectories that exhibit the emergent behavior of interest. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed method is interesting. It appears reasonably general (provided that a function measuring the behavior of interest can be defined, and the environment scenarios of interest parameterized) and potentially a very useful way to enhance understanding of robot controllers beyond measuring task completion.\n    * Of particular value is the focus on representative trajectories.\n2. The experimental evidence provided is interesting (taking into account the appendix), especially Table 1 which highlights the differences between controllers. This type of controller characterization provides a rich and valuable way of understanding controllers that is complementary to task completion metrics. \n3. The problem of characterizing and understanding emergent robot controller behavior is highly relevant to robot learning, especially if learning robot controllers are to be deployed in the real world. The problem setting is well motivated and explained in the paper. \n4. The related work is thorough.\n5. The paper is well written, the clarity of presentation is high throughout\n6. The video is a valuable addition, especially the chart presented starting at 45s\n7. The behavior taxonomy is valuable, well described, and appear comprehensive\n    * Potentially consider looking into obstacle clearance velocity (i.e. average speed at closest distance to obstacle) in future work.\n8. Sufficient detail appears to be provided to reproduce the work taking into account the appendix. The code is also released.\n\n**Weaknesses**\n\nNo baseline\n* The main alternative to RoCUS presented in the paper is a simple top-k baseline however no results using this approach are presented. This makes it difficult to assess the benefits of RoCUS over the (much simpler) baseline.\n\nScope\n* Whilst this paper is relevant to robot learning in that it proposes a method for better understanding complex and potentially high dimensional learned robot controllers, there is no evidence provided or discussion of how the results in simulation are relevant to physical robot controllers. Potential questions of interest include:\n    1. Can this method be applied to a physical robot controller? If not, why not?\n    2. If this method is only applicable in simulation, can it be used to understand physical robot controllers? If not, why not and how might this limitation be overcome in the future?\n\nThe results are a little challenging to interpret.\n* Figure 5 and Table 1(appendix) indicates that there is significant variation between the different controllers’ emergent behavior. This is interesting, but it is challenging to interpret and compare controllers because the likelihood of the posterior samples is not clear. Is the likelihood of the posterior samples equal across all three controllers? It would be even more useful if the likelihood could be provided for each controller and behavior for all posterior means.\n* To give an example, consider Table 1(appendix), Behavior: Length. The IL and DS controllers have a similar posterior mean length (161, 166 respectively), but quite different Prior Means (161, 201 respectively). It is not clear how to compare IL and DS on trajectory length for representative samples. Are trajectories with sample length of 161 and 166 equally likely for IL and DS?\n\nA number of the most interesting results are located in the appendix, and the main paper’s experimental evidence is not as compelling without it. \n* Suggest moving Table 1 and Table 2 to the main paper.\n* Suggest moving the discussion of the stochastic controller to the appendix to make space. Similarly the explanation of the RRT controller could be shortened since controller understanding instead of design is the main focus of the paper\n\nMinor\n* It is not clear why IL is used in one experimental setting (2D nav) and RL in another (Reaching)\n* The sampling formulation (see eq 2) appears to be very similar to BAYES-TREX. It would be nice to see a brief discussion in the related work describing how RoCUS differs (e.g. trajectories vs classification, MCMC vs MH sampling).\n* Notation: The use of No() for the Normal distribution is confusing since it is non-standard. The notation in [12] is clearer.\n* Typo: 172 I think this should read (Line 7) instead of (Line 6).\n* It is not clear why all the emergent behaviors are not evaluated for the 7DoF reaching task (Table 2) whilst they were for 2D nav (Table 1). It would be valuable to see Table 2 extended to include all of the behaviors from Table 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Thank you for this submission. Overall this is an interesting method that offers insight into an important problem in robot learning (understanding emergent controller behavior). The method is well explained and the results in simulation are interesting, especially when the appendix is taken into account. It is also a pleasure to read. However there are a few issues, described above, some of which are significant.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "* Please consider the four main issues of baseline, scope, results interpretation, and structure\n* Please also address the minor points listed above.\n", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to analyze the "emergent" behaviors of a controller, assuming that the behavior metric can be defined. All reviewers agree that the paper addresses an important problem in robotics. 

However, to fully appreciate the contributions of the proposed method, the authors should qualitatively and quantitatively compare it against other baselines. The authors should also clearly discuss the challenges, if any, in deploying the proposed approach on real robots. 

Besides addressing the major issues mentioned above, the authors should also revise the manuscript according to the other clarifications requested and the suggestions provided.

===== Post rebuttal ===== 

The authors have sufficiently addressed reviewers' concerns; specifically, the comparison with additional baselines was particularly insightful. I recommend an acceptance. 
Decision: Accept (Poster)