Quantifying Assistive Robustness Via the Natural-Adversarial Frontier
Authors: Jerry Zhi-Yang He
Daniel S. Brown
Zackory Erickson
Anca Dragan
Venue: CoRL 2023 Poster (2023)
Keywords: assistive robots
safety
human-robot interaction
adversarial robustness

Abstract: Our ultimate goal is to build robust policies for robots that assist people. What makes this hard is that people can behave unexpectedly at test time, potentially interacting with the robot outside its training distribution and leading to failures. Even just measuring robustness is a challenge. Adversarial perturbations are the default, but they can paint the wrong picture: they can correspond to human motions that are unlikely to occur during natural interactions with people. A robot policy might fail under small adversarial perturbations but work under large natural perturbations. We propose that capturing robustness in these interactive settings requires constructing and analyzing the entire natural-adversarial frontier: the Pareto-frontier of human policies that are the best trade-offs between naturalness and low robot performance. We introduce RIGID, a method for constructing this frontier by training adversarial human policies that trade off between minimizing robot reward and acting human-like (as measured by a discriminator). On an Assistive Gym task, we use RIGID to analyze the performance of standard collaborative RL, as well as the performance of existing methods meant to increase robustness. We also compare the frontier RIGID identifies with the failures identified in expert adversarial interaction, and with naturally-occurring failures during user interaction. Overall, we find evidence that RIGID can provide a meaningful measure of robustness predictive of deployment performance, and uncover failure cases that are difficult to find manually.

Quantifying Assistive Robustness Via the Natural-Adversarial Frontier
Jerry Zhi-Yang Hel, Zackory Erickson", Daniel S. Brown', and Anca D. Dragan'
University of California Berkeley, CCarnegie Mellon University, University of Utah, anca bberkeeyy edu, dsbrown@cs. wwwp:/eeesssu.hh.....omom edu, zerickso@cmu. edu [hayjerry,anca]
Abstract: Our ultimate goal is to build robust policies for robots that assist people. What makes this hard that people can behave unexpectedly
time, potentially interacting with the robot outside its training distribution and leading to failures. Even just measuring robustness is a challenge. Adversarial perturbations are the default, but they can paint the wrong picture: they can correspond to human motions that are unlikely to occur during natural interactions with people. A robot policy might fail under small adversarial perturbations but work under large natural perturbations. We propose that capturing robustness in these interactive settings requires constructing and analyzing the entire natural-ade natural-adversarial frontier: the Pareto-frontier human policies that are the best trade-offs between naturalness and low robot performance. We introduce RIGID, method for constructing this frontier by training adversarial human policies that trade off between minimizing robot reward and acting human-like (as measured by discriminator). On Assistive Gym task, we use RIGID to analyze the performance of standard collaborative Reinforcement Learning, well the performance of existing methods meant to increase robustness. We also compare the frontier RIGID identifies with the failures identified in expert adversarial interaction, and with failures during user interaction. Overall, we find evidence that RIGID can provide meaningful measure of robustness predictive of deployment performance, and uncover failure cases in human-robot interaction that are difficult to find manually. https://ood- http:/anod.hmn......m gi thub. io
Keywords: assistive robots, safety, human--rboo interaction, adversarial robustness
1 Introduction
Deploying assistive robots in healthcare facilities and homes hinges crucially on their robustness, reliability, and the assurance that they can safely interact with individuals of all ages, from children to older adults. The worst-case scenario would involve an errant robotic arm causing injuries. However, slight variations or unpredictable elements in the operational environment can influence the behavior of learned robotic policies. Consequently, we must thoroughly examine the robot's 's response not just to regular situations but also to potential scenarios where unexpected events may unfold.
The complexities of stress testing robot policies, however, can be exceptionally high. For instance, verifying the robustness of dish-loading robot in all kitchen configurations is highly costly [1]. Verifying human-robot interaction applications, in comparison, is even more challenging due to the fact that the human partner performs independent actions, changing the state that the robot's 's policy responds to, and potentially inducing distribution shift.
The common approach to probing the policy's robustness - perturin inputs to the model would translate here to perturbing the human partner's actions. But this can paint a misleading picture of the robustness of assistive policy: while in theory any human action is possible, in practice humans won't move in arbitrary ways. A robot policy might thus fail under small adversarial perturbations but work under quite large natural deviations in human motion.
We thus propose to measure robustness to natural variations in human motion. This begs the question of how to define "natural". We propose a generative approach: given a dataset of motions
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Adverarialnes
Adversarial human motions
"Seemingly natural yet adversarial"
Natural human motions
Naturalness
Robot failure
Robot failure !
Robot success
Figure I: We propose measuring assistive robustness by considering the naturalness of human motions, and analyzing the entire natural-adversarial frontier: points with natural human motion that lead to good robot performance (left), unnatural motions that easily break the policy (right), and natural motions that still lead failures (center). The full frontier paints more useful picture of robustness can be reduced to a scalar using the area under the curve (AUC): lower AUC is indicative of higher robustness.
looking a single point, and
demonstrated by the human in the target task sampled from trusted human model/simulator [2, 3] we train a GAN [4, 5] and definite naturalness via the learned discriminator.
Next comes the question of how unnatural we allow human motions to be in our search for adversarial attacks against the robot: it easy to break most policies if we allow motions that are very unnatural, and we might not find the interesting failure cases if we restrict the motion to be too close to what was seen in training. We thus advocate that robustness should not be assessed by looking at a single threshold, but rather by looking at the entire natural-adee adversaraa frontier.
Our proposal is measure assistive robustness by considering the naturalness of human motions and analyzing the entire natural-adversaae frontier.
This Parcto frontier likely contains points poor robot performance caused by unnatural human motions and points of good robot performance interacting with naturally moving humans. What is interesting lies in between: there exist natural motions that can lead to unexpected failures as in Fig. leading to high area under the curve.
We introduce RIGID, method that constructs the adversaaaaa frontier by training adversarial human policies that trade-off between minimizing robot reward and acting in a human-like way, as measured by the discriminator 1. construct the frontiers for different policies - one trained to naively collaborate with the human, and others trained with robust Reinforcement Learning methods (c.g. by using populations) [6. 7]. While we find that prior robust RL methods improve policy robustness measured by RIGID, we are able to uncover edge cases where natural motions (natural with regard to our simulated human) cause robot failures. We then use RIGID to improve robustness by fine-tuning regular RL using data points identified by RIGID. Finally, we conduct user study with naive users and an expert who attempts to lower the robot's reward. We find that RIGID can generate failure cases more effectively than manual effort, and is predictive of deployment performance.
2 Related Work
Adversarial attacks and robustness have long been studied in machine learning, with notable examples in computer vision [8, 9, 10, 11, 12, 13). Researchers have successfully compromised state-of-the-an computer vision models with simple manipulations of pixels [14, 15) imperceptible human eyes, which leads to real-world safety concerns. Since then, the community has persistently come up with many attacks [16, 17, 18] and defense mechanisms [19, 20, 11]. Most of the attack methods can be characterized by their allowed perturbation set, such as bounded 12-norm, and the method of optimizing adversarial examples [21]. Robustness methods, on the other hand, often require training against adversarial examples by adding them to the training data [9, 22, 23] as a way of performing robust optimization [24] or focus on regularization and smoothing techniques [25, 20, 26, 11].
Robust control [27] in robotics focuses on deriving safety guarantees in the presence of bounded modeling errors and disturbances [28, 29, 30, 31]. It has been applied to human-robot interaction
While in our work we use simulated human to generate data for the discriminator, in general, this data can also come from human- human or human-robot interaction in the task.

---

by allowing completely adversarial human motions (the full forward reachable set), as well as restriction to only motions that are sufficiently likely under a human model [32]. Robustness in reinforcement learning is growing field inspired by the brittleness of RI. policies [33, 34). Prior work on adversarial attacks focuses on single-agent [35] or two-player competitive settings [36, 37,38, 39]. In collaborative settings, policies are known to be more brittle [2] than zero-sum games. Recent works have also looked at improving the generalizability in collaborative settings, which is a form of robustness against naturally occurring human distributions [6, 40, 41]. Our work is complementary, providing a way to measure robustness, and perhaps paving the way to new methods to improve it.
Dataset Aggregation (DAgger) [42] is framework for passively incorporating failure cases into robot learning. One limitation of DAgger is that requires online interaction with real humans in order to improve robustness- - failing, and having the human correct or otherwise augment the data and retraining. This is undesirable in assistance and may cause harm to users. In contrast, our framework is an instance of active learning [43] that looks for failure cases of the robot policy before deployment. This way, we actively discover failure cases in simulation without worl-world consequences.
Generative adversarial imitation learning (GAIL) [5] is promising method for producing motions that mimic a set of demonstrations by training a discriminator on dataset to provide the reward for a reinforcement learning agent. Researchers have experimented with variants of GAIL and found that it leads to high-quality motions in graphics [44], locomotion [45], and even controllable natural motions [46]. In this work, we build on the existing techniques for training GAIL to generate human motions that are perceived as natural. This allows us to study assistive robustness: by optimizing for
the joint objective of being both natural and adversarial, we search for range of plausible and typical human motions that create catastrophic failures for robot policies in assistive settings.
3 Natural-Adversarial Robustness
In this section, we first introduce the robotic assistance problem in Fig. 2, where the goal is assist a human in a partially observable setting - the human may have hidden information, such as intent or preferences, that needs be inferred on the fly. We then formulate the problem of adversarial but natural human motion, and reduce it to the optimization of an objective trading off between the two.
Assistance as Two-Player Dec- POMDP. Following prior work [47, 48, 6] we model an assistive task as two-agent, finite horizon decentralized partially-observabbe Markov decision process (Dec-POMDP), defined by the tuple (S, ar, AR, AH. T, 22R Q2. O, R). Here S is the state space and Ar and Ah are the human's ' and the robot's action spaces, respectively. The human and the robot share a real-valued reward function R : S x Ar x AH - R; however, we assume that the reward function is not fully observed by the robot, i.e., some of the reward function parameters (e.g., the specific goal location or objective of an assistive task) are in the hidden part of the state. T is the transition function where T(s' | s,aR, ( is the probability of transitioning from state to state s' given ag E AR and an E Ah, Qe and QH are the sets of observations for the robot and human, respectively, and O S' AR X Ah - QR x Qu represents the observation probabilities. We denote the horizon of the task by T.
Figure 2: Assistive task: itch scratching with un- known itch locations. See appendix Sec. B for more details.
Adversarial Human Motions. Let human trajectory denote a sequence of states and actions T= = (f) ST. ST.... aaf). and let function ли be the human policy that maps from local histories of observations off = (off .fff over QH to probability distribution over actions. We define for an arbitrary human policy ли. its corresponding assistive robot policy TR(H). which is the robot policy that is optimized to collaborate best with лh with respect to the joint reward R: TR(H) = aragrmaa m, [Σ, [T)).
We can then define an ad adversarial human policy म& with respect to an assistive robot policy. The policy मे minimizes the overall performance under the constraint that 7h similar to the original policy TH² 3 as measured by an f-divergence measure Dy (7 (|TH) ) S 6. and adjustable coefficient ò
2For the sake of simplicity, we use a modified R to denote the expected reward of the rollouts of human-rohot policy pairs as R(TR.TT) := Baf-n,((s)) Ba - (st)),,,)) [Σ, .at,aa,),
(s)) *Typically, robot policies are trained via RL as best response to human model [2]: offline RL [49] is also possible and bypasses human model, in which case the discriminator can be trained directly the offline data.
3

---

controls the allowable perturbation set in which the adversarial human policy Aи deviates from THe similar adversarial perturbations in computer vision [11]: Rh (TR(M))) d) argmin| R ((T)))) s.t. Dy (Tzz) (T) S
(1)
"i
म and N are typically modeled by neural networks, making difficult to directly compute their differences. We hereby introduce methods to compute their f-divergence via resulting trajectories.
Approximating the Divergence Measures such as KL divergence [50, 51, 3, 52], x² [53] divergence are commonly used to characterize the distance between two probability distributions. In the context of policy learning, this can be achieved by training a discriminator 2 T E to distinguish between trajectories sampled from Aи and πи- More specifically, D assigns score D(T) == E (D [T)] to any policy π. It is trained to assign low scores to trajectories drawn from the true human policy тH and high scores to trajectories drawn from the ad adversarial policy 7n. We choose the LS-GAN objective [53] for training D:
D = arg min BT-
((xp(((()++))
(2)
D
Previous work [53] has proven that this represents x² divergence when trained to optimality: D Dys( = Br [. We illustrate this in appendix Sec. H. Note that while we focous on
x" divergence in this paper, in practice one may use other divergence measures such as KI. divergence and such as MMD (Maximum Mean Discrepancy) [54]. See appendix Sec. H for additional results.
Adversarial Frontier While previous work in Computer Vision (CV) [12, 11] assign the perturbation set to a fixed value, it is unclear what value we should assign to 5 in the context of Halman-Robou Interactions. Instead, we consider scanning over all possible values of ō to find the adversarial human policy under all different naturalness criteria. In other words, we consider all levels of "naturalness" in human motions. We consider the Lagrange dual function of the constrained optimization in Eq. (I):
L(RR, = max A
R( R(TRATT xH - 1 D (T's (| + 6
(3)
RH (TR. = arg max |- R(TR.TTH) Z/z
(4)
We have reduced the constrained optimization in Eq. (1) to an unconstrained optimization in Eq. (4) that balances "adversarialness" (playing an adversary to the robot by optimizing for - and "naturalness" (staying close to the canonical interaction by minimizing a divergence metric D()). The parameter A provides a knob that we can use to trade off how adversarial and how natural we would like Āи to be. Setting high reward. On the other hand, setting A = O leads to a policy 7н that is purely adversarial and causes harm the assistive task by inverting the environment reward. By selecting the naturalness parameter A E [0, co), we arrive spectrum of human motions that interpolate between adversarial and natural. This is meaningful in Sluman-Robo Interaction because while we may not need to worry about purely adversarial human behaviors, as they are less likely to happen in reality, we need to take precaution against human behavior that appears natural, yet causes robot failures.
results in a policy 7и that closely resembles ии and yields a
Given these definitions, we can in principle compute the naturalness and adversarialness of all possible human motions, and plot them 2D coordinate system. This motivates the Natural-Adversarr curve - the Pareto frontier of non-dominated policies. These are policies that, compared to any other ones, have either better adversarial score, or better naturalness score, or both.
4 Computing the Natural-Adversarias Frontier
In this section, we propose a practical method that we call RIGID that can help obtain the Adversarial Frontier in Sec. 3. Having motivated the concept of an adversarial human policy R (R) and the naturalness parameter, A, in Sec. 3, now discuss the Vatural-Adeessarial Pareto frontier (Sec. 4.1). We then introduce an algorithm, RIGID (Sec. 4.2), that efficiently approximates the natural-adversarial curve.
4.1 Connecting The Dots: the Natural-Adversarial Curve
Points Along the Frontier: To solve for Eq. (2), we start with sampled trajectories from TH (in our experiments, a human model; in general, this can also be data collected from human-human or

---

Adversrial Incss lncss
Adversariaines
Step I(1)
Step I(2)
Typical
curve
Step 2(1)
Step 2(2)
Step 1(1)
Step ((2)
Step 1(3)
Adversatialng
Step 2(3)
Safe
Am 0
Step 201)
Step 212)
Step 213)
Step
(C)
Naturalness
Naturalness
Figure 3: Left: different types of Natural-AAversarial curves. (b) has a higher AUC means it is less robust and more prone to unsafe behaviors. The idcal (c) curve looks like sharp drop. Right: when running RIGID in Alg. I, we first sample X's evenly in the log space over A A.min.Amat in (d). We then iteratively refine picks by sampling more finely over selected intervals to gain better coverage of the curve (c).
human-robot interaction). For specific A, we adapt GAIL [5] to our setting: we interleave training the adversarial policy æи and the discriminator D by iteratively (a) doing one step of policy optimization on Eq. (4) to adapt the policy and (b) doing one gradient step on Eq. (2) to adapt the discriminator, In practice, we use PPO [55, 56] as our policy optimization algorithm, and we use LS-GAN [53] with noise annealing to ensure the stability of the training process.
To plot the resulting An in the natural-adversarial coordinate, we compute naturalness (x coordinate) by sampling trajectories t ~ R. and calculate the mean likelihood of D(T) < O. This is equivalent to having the discriminator classify the trajectories from the original human model. We define adversarialness (y coordinate) as negative robot reward normalized to [0, IJ. See appendix Sec. D.
Approximating the Curve: Once sample a sufficient number of As (see Sec. 4.2 on how to do this) from (0,,o), we con- nect the outermost points. The resulting curve characterizes the frontier of adversar- ial human motions under different levels of naturalness constraints (Eq. (4)) or different perturbation sets (Eq. (1)). Intu- itively, under the naturalness value, lower ad versarialness corresponds to more robust robot policies. Fig. shows three different hypothctical curves, visualizing how the shape of the curve connects the robustness of the policy.
Algorithm RIGID
Require:
1: Maximum refinements d, samples per round k Upper and lower bounds on A: Amax dmin procedure RIGID (A Amax, d)
Aal = [], Snat = []. Sadv = []
4:
1' Amin d'mxx = Amax
for I....,d do
6:
Find evenly spaced el...e in lo log 1 X
7:
for j l.....k do
exp(ep())
9:
Aull-append(2))
10:
Compute सи using Eq. (4) and dj
The AUC Score
Scalar Robust-
11:
Compute nat, and adv, for An (dz)
12: Sua. Spa.aap nat,), Sadv apt.ndpped
ness Metric: Given a Natural-Adversarial curve, we compute the Area Under Curve
13: end for
14: A' min" N'ax LARGEST-JUMPCS pat
(AUC) to quantitatively measure robust-
15: end for
ness as a single scalar. Lower AUC gener- ally means stronger robustness, although there may be outlier human policies that can trigger unsafe behaviors, and a full plot of the curve still necessary.
16: return Spats Sadv
end procedure
4.2 Sampling Useful Trade-Offs
When we gradually increase A, the resulting adversarial human motion often goes through mode changes, which lead sudden increases in naturalness. Motivated by this, we need to select [0, (0) in non-uniform manner, by procedurally expanding between the two 2's where naturalness increases
5

---

Rabot Policy
Robot Palicy
0.8
Adversrianees
0.6
Co.
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1,00
Naturaluess
Naturalness
Figure 4: Natural-Adversarial curves of two different robotic policies trained using vanilla RL, with random seed differences. Every point corresponds to motion policy found by RIGID for a particular 2 value. We highlight the frontier in orange. The visualization shows how points on the frontier correspond to failure cases.
the most. We do this through an iterative refinement approach by finding the largest gap in naturalness and increasing the sampling density there.
Our algorithm's 's pseudocode is shown in Alg. 1. The lists Smar and Sndy keep track of the corresponding naturalness and adversarialness values of points along the Pareto frontier. We initialize Amin > 0 be the smallest A we consider, and Amax a sufficiently large number. We denote all the a considered so far as Aall- The procedure LARGEST-JUMP looks at all selected so far and selects two values between which the naturalness changed the most (see appendix Sec. A for the pseudo-code).
5
Experiments
5.1 How robust is Vanilla RI.?
In this section, we use RIGID to analyze vanilla RL robot policies and to uncover possible natural (according to our simulator) motions that trigger failures. We then examine whether the RIGID frontiers are predictive of deployment performance: we test these policies with (a) end users and (b) an expert who adversarially attempts to get the robot to fail. We then plot the resulting behaviors relative to the RIGID-computed frontier. We find that RIGID is able to find points that are just as natural, but more adversarial Next, we analyze existing algorithms for robustifying RL policies and compare them according to RIGID. We find that RIGID does help differentiate between more and less robust policies. We also show that training with the natural adversarial human motions identified by RIGID leads to more robust RL policies. We leave implementation details, computational complexity. and data requirements to Appendix Sec.
The Assistive Itch-Scratching Task [57] is visualized Fig. 2. The human is seated in a natural pose with seat-mounted robot arm. Only the human knows the position of the itch location their arm. Both agents are rewarded for seratching the itch location and penalized for contacting anywhere else on the human body. Success is achieved if the robot achieves meaningful contact for 25 timesteps.
First, we study whether assistive robot policies trained with SOTA reinforcement learning algorithms robust. To do this, we first generate synthetic human policies by jointly optimizing for human and robot policies following [58]. We keep and freeze the resulting human policy as the synthetic human that use, and train assistive robot policy using the PPO algorithm to assist the synthetic human. The resulting robot policies achieve nearly 100% success rate with the training human model. We then use RIGID come up with natural and adversarial human policies attack this robot policy. See appendix Sec. D for further details.
In Fig. 4, we visualize the resulting human policies from the RIGID algorithm points and connect the dots on the outermost boundary to form a -aderal-Adversariaal frontier. Visualizations of points with naturalness values in the range [0.4, 0.8] show abundant failure cases - the adversarial human policy moves naturally (according to the model), yet causes the robot to fail. We show two examples in Fig. 4 with the dashed arrow. See appendix Sec. D for more visualizations.
'The caveat is that naturalness here is only as good as the synthetic human models we used in the experiments. This could be improved by using real-world numan-human human-robot interaction data

---

Robot Policy
Robot Policy 2
Percentage of time with higher ratings
1.0
80%
Adv. Expert 70%
0.8
Natural User
Adversorialnss
60%-
0.6
50%
0.4
40%
0.2
30%
0.0
0.2
0.44
0.6
0..8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
20%
Naturalness
Naturalness
RIGID
Natural
Figure 5: The RIGID-identified frontier for two policies. We find that RIGID pareto-dominates natural user and adversarial expert behaviors. When taking two behaviors with the same naturalness score user's and RIGID's RIGID finds more adversarial solutions. On the right, we see that users indeed do not find large difference between these two behaviors "naturalness" (similarity human training data).
While we can verify that points on the Natural-Adversaria curve are indeed failure cases of the robot policy, one may still question whether this curve is exhaustive and faithful. Here, exhaustive means that this curve encloses all the possible failure cases that one can produce, and faithful means that the trajectories deemed natural by the curve are judged as natural the same extent by human beings.
Exhaustiveness We perform a woopprrt user study based on a virtual reality (VR) assistive gym plugin [59] to verify exhaustiveness. First, recruit eight novice users who do not have prior experience interacting with the robot in VR. We inform them about the task, the objective and physical constraints and have them watch few episodes of successful interactions. The robot then assists the users, simulating regular daily deployment conditions around normal users. Second, we stress-test the system emulate what might happen under a wider range of users and over prolonged deployment. We have an expert (one of the authors) act adversarially to cause the robot to fail (collide). the curve produced by the RIGID algorithm is exhaustive, trajectories from both the regular users as well as the expert should lie underneath the curve, which is indeed the case (Fig. 5): points on the RIGID curve commnntdominate benign user and even expert adversarial data. These results suggest that RIGID is more effective at finding edge cases than manual efforts.
We also that for these two policies, the AUC is smaller for the second, robust. Anecdotally, the expert reported that the first policy was easier to break by simply extending their arm, which sent the policy into an unstable rotating behavior. This suggests that RIGID correctly identified which policy is more robust.
expect it to be more
Faithfulness To test whether the curves are faithful, we ask 15 novice users to judge whether trajectory pairs of similar "naturalness" generated by RIGID and the previous users are qualitatively similar. We show users trajectories from the original human-robot pair. We then ask them to compare RIGID and user trajectories and rate which motion closer to the reference data. The plot on the right of Fig. 5 confirms that the "naturalness" metric in RIGID corresponds to the user judgments.
5.2 Do robust RL methods improve robustness? Can training with RIGID examples help?
In this section, we leverage RIGID to efficiently find Natural-AAverssrrall curves for different robust robot learning methods, allowing us to compare their assistive robustness.
In Fig. 5.2, we compare the following methods: (1) Gleeve et al [7] use human-policy randomization during the co-optimization phase to improve the robustness of the robot, (2) PALM [6] leverages distribution of different human behaviors to Iearn a latent representation that enables better generalizations. (3) We also study an oracle method called Robust GT, where we fine-tune Vanilla RL on failure cases from the Natural-Adversan population, and resuming training from the Vanilla RL checkpoint.
curve, by adding the failure cases to the training human
We notice that while existing robustness methods improve over Vanilla RL in terms of Assistive Robustness AUC, they have visible failure cases (see appendix Sec. F). We also evaluate the performance of different robust policies on assisting humans. The robust optimization method from Gleeve et [7] improves AUC at the cost of worse performance when assisting more natural human policies. For PALM [6], we also observe this performannc--rbuustees trade-off. We find that Robust
7

---

GT, which is trained on adversarial human policies found from RIGID achieves the best AUC reduction while also achieving good performance when assisting natural human policies.
Method
AUC (L)
Success (1)
Vanilla RL
0.630
1.0
0.8
Gleeve et al [7]
0.584
0.83
Adversarialness
0.6
PALM [6]
0.668
0.95
Robust GT (Ours)
0.473
1.0
0.4
Vanilla RL
Figure 6: Left: Natural-Adversarial curves of differ- ent methods. Right: We compute the Area Under Curve (AUC) of the Vatural-Adversarial curves and the success rate of robot policies trained with different methods.
0.2
PALM
Gleeve ct
Robust GT (Ours)
0.0
0.2
0.4
0.6
0.8
1.0
Naturalness
Limitations
One limitation that the Natural-AAversarial curve carries some randomness. While the frontier exists, in practice one has to rely on multi-objective optimizations which have inherent randomness and sub-optimality. As a result, we can only approximate the frontier. Performing RIGID for more iterations under a diverse set of Naturalness objectives can help reduce this error.
The second limitation that in our experiments, we rely on synthetic humans generated by human- robot co-optimizations [58, 57]. While such human agents perform reasonable movements (see visualizations in appendix Sec. B) and we have conducted extensive user studies in Sec. 5.1, future work should incorporate real-world human data to create more human-like simulated agents using Behavior Cloning or Offline RL. [49]. It is also worth noting that in our studies, we use data from simulated humans as the canonical datasets on which we train GAN and measure "naturalness". While this may seem limiting, as we have only tested on our simulated humans, the framework is applicable to human-robot applications where there typically exists canonical way of interaction (i.e. dressing requires the human to extend their arm to initiate the contact).
A third limitation is that in our implementation of Eq. (2), we use a memory-less discriminator. Because of this, the adversarial behaviors we find are primarily state-based. Using temporal naturalness measure can help us discover more subtle temporal ad adversarial behaviors. Training such temporal measures, however, imposes new challenges on policy learning and is beyond the scope of our focus.
Last but not least, our proposed algorithm RIGID relies on batch training of human adversarial policies. On the one hand, this means that in environments where policy learning is difficult (i.e. sparse reward), our framework is not applicable. On the other hand, computational complexity can be challenging. RIGID only demonstrates the feasibility of computing the Natural-Adversarial curve, and we believe that there are a number of ways to speed up RIGID. This includes Quality-Diversity methods [60, 61], using Evolutionary Methods to perturb one policy's 's rollouts, and fine-tuning one base adversarial policy instead of training all from scratch are promising directions.
7 Conclusions
We propose Assistive Robustness measure for evaluating the robustness of robot policies in assistive settings. Verified by user studies, we show that the Vatural-Adversarii curve effectively represents what extent the robot remains safe under plausible human perturbations. Our proposed algorithm RIGID is an initial attempt at effectively computing the curve. While we study physical interaction in healthcare as the main application this work, we believe our framework can apply to general Human-Robot collaborative settings. We are excited for proposing framework and tools to facilitate research safety in assistive and collaborative settings, more sophisticated robotic applications are being deployed in the real world.
8

---

Acknowledgments
We would like to thank Cassidy Laidlaw and Erik Jones for helpful discussions on adversarial perturbations. This research was supported by the NSF National Robotics Initiative.
References
[1] R. Tedrake. Tri taking the hard problems in manipulation research toward making human-assist robots reliable and robust, URL.
[2] M. Carroll, R. Shah, M. K. Ho, T. L. Griffiths, S. A. Seshia. P. Abbeel, and A. D. Dragan. On the utility of learning about humans for human-ai coordination. CoRR, abs/1910.05789, 2019. URL htth: //atx:v/ ottp//ii.orr.ooooooooo. abs/1910 05789.
[3] C. Laidlaw and A. Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models, 2022.
[4] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, and Y. Bengio. Generative adversarial networks, 1-9, arXiv preprint arXiv: 1406.266,, 2014.
[5] Ho and Ermon. Generative adversarial imitation learning, 2016.
[6] Z.-Y. He, Erickson, D. S. Brown, A. Raghumathan, and A. Dragan. Learning representations that enable generalization in assistive tasks. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 Proceedings of Machine Learning Research, pages 2105-2114. PMLR, 14-18 Dec 2023.
[7] P. Czempin and A. Gleave. Reducing exploitability with population based training. In New Frontiers in Adversarial Machine Learning Workshop ICML, 2022.
[8] C. Szegedy. W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, Goodfellow, and Fergus. Intriguing properties of neural networks. arXiv preprint /3/.6666/ 2013.
[9] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXi::1412.6572, 2014.
[10] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99-112. Chapman and Hall/CRC, 2018.
[11] J. Cohen, F.. Rosenfeld, and Z.. Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pages 1310-1320. PMLR, 2019.
[12] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 2019.
[13] Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pages PMLR, 2020.
[14] Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transferable adversarial examples and bla attacks. arXiv preprint arXiv: 2016.
[15] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint 170.:1705.07204 2017.
[16] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel. Adversarial attacks on neural network policies. arXiv, 2017. URL https://arxx:
hatg//bs/77000 1t////0222..2...
[17] J. Uesato, B. O'donoghue, P. Kohli, and A. Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pages 5025-5034. PMLR, 2018
[18] A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. International conference on machine learning, pages 274-283. PMLR, 2018.

---

[19] Y. Song, Т. Kim, S. Nowozin, S. Ermon, and N. Kushman, Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. CoRR, abs/1710.10766, 2017. URL Utl.http://arxiv.orr org/ abs/ 11100 10766.
[20] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pages 854-863, PMLR, 2017.
[21] Z. Kolter and A. Madry. Adversarial robustness - theory and practice, URL.
[22] T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, and S. Ishii Distributional smoothing with virtual adversarial training. arXiv preprint arXiv:1507.000 2015.
[23] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. arXiv preprint 1r1iv:161..01220 2016.
[24] U. Shaham, Y. Yamada, and S. Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. arXiv preprint arXiv:1511.05422 2015.
[25] S. Gu and L. Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint 142:5.14 2014.
[26] A. Raghunathan, J. Steinhardt, and P. Liang. Certified defenses against adversarial examples. arXiv preprint 1X01:/800009944 2018.
[27] A. Bemporad and M. Morari. Robust model predictive control: A survey. In Robustness in identification and control, pages 207-226. Springer. 2007.
[28] J.-J. Slotine. The robust control of robot manipulators. The International Journal of Robotics Research, 4(2):49-64 -4, 1985.
[29] G. Zames. Feedback and optimal sensitivity: Model reference transformations, multiplicative seminorms, and approximate inverses. IEEE Transactions on Automatic Control, 2612)3313.223 1981. doi: 10. 109/TAC. 1981 100260.
[30] M. Zeinali and I.. Notash. Adaptive sliding mode control with uncertainty estimator for robot manipulators. Mechanism and Machine Theory, 45(1):80-90, 2010. ISSN 0094- 114X. doi:
wttp::/o..r/1111......1.11000. wttgg//...01/...mmeee.ee http:///........//mmmmmmoooooo.p .8..03.. URL htthttps::/ sciencedirect. com/science/ httpr//eeeeerrteeeee//////o/////
[31] E. Sariyildiz, R. Oboe, and K. Ohnishi. Disturbance observer-based robust control and its applications: 35th anniversary overview. IEEE Transactions Industrial Electronics, 67(3): 2042-2053, 2019.
[32] A. Bajesy, S. Bansal, E. Ratner, C. Tomlin, and A. D. Dragan. A robust control framework for human motion prediction. IEEE Robotics and Automation Letters, 6(1):24-31, 2020.
[33] C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A study overfitting in deep reinforcement learning. arXiv preprint 184.093804.06633, 2018
[34] A. Gleave, M. Dennis, N. Kant, C. Wild, S. Levine, and S. Russell. Adversarial policies:: Attacking deep reinforcement learning. CoRR, abs/1905.10615 2019. URL http:///xxx org/abs/ /t/aa/bs/199... 1900/ 10615.
[35] I.. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. CoRR, /tbs//703.02702. 2017. Utt..ttp://axxii..orr//b/////
02702..
[36] S. H. Huang, N. Papernot, J. Goodfellow, Y. Duan, and P. Abbeel. Adversarial attacks neural network policies. CoRR, abs/1702.00284, 2017. URL utt.ttp://arxivv.orr///e a100///..//0//./..... 02284.
[37] A. Mandlekar, Zhu, A. Garg, L. Fei-Fei, and S. Savarese. Adversarially robust policy learning: Active construction of thysically-plausiin perturbations. In 20/7 IEFE/RSI International Conference Intelligent Robots and Systems (IROS), pages 3932-3939. IEEE, 2017.
10

---

[38] K. Mo, W. Tang, J. Li, and X. Yuan. Attacking deep reinforcement learning with decoupled adversarial policy. IEEE Transactions on Dependable and Secure Computing, 2022.
[39] T.. Wang, A. Gleave, N. Belrose, T. Tseng, Miller, M. D. Dennis, Y. Duan, V. Pogrebniak, S. Levine, and S. Russell. Adversarial polisrieepooiiiess beat professional-level go ais. arXiv:221 1.00241v [cs.LG], 2022.
[40] D. Strouse, K. R. McKee, M. M. Botvinick, E. Hughes, and Everett. Collaborating with humans without human data. CoRR, 0oss/210.081777 2021. URL https://arxiv org/abs/ 2110.0811
[41] A. Lupu, B. Cui, H. Hu, and Foerster. Trajectory diversity for zero-shot coordination. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 7204-7213. PMLR, 18-24 Jul 2021. URL https: / http:///oooeeeeiss./////// mlr wpwess/vvvvvv.... (tt1119/uppppe 1a. html.
[42] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[43] J. Z. He and A. D. Dragan. Assisted robust reward design. CoRR, abs/21 /bs/2111.09884. 2021. URL https:// ptaps://arxiinn org/ abs/2111. 09884..
[44] X. B. Peng, Abbeel, S. Levine, and M. van Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. CoRR, abs/1844000777,, 2018. 2018. URL http://arxiv, org //s abs/aasss../..///////////// 1104111440000 02717.
[45] A. Escontrela, X. Peng, Yu, Zhang, A. Iscen, K. Goldberg, and Abbeel. Adversarial motion priors make good substitutes for complex reward functions, 2022.
[46] X. B. Peng. Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. AMP: adversarial motion priors for stylized physics-based character control. CoRR, abs/2104.02188 180, 2021. URL https://arxiv. org/ abs/2 220.//22...00 2104.02180.
[47] D. Hadnield-Menell, S. J. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning, Advances in neural information processing systems, 29, 2016.
[48] Taha, J. V. Miró, and G. Dissanayake. A pomdp framework for modelling human interaction with assistive robots. In 2011 IEÉE International Conference on Robotics and Automation, pages 544-549, IEEE, 2011.
[49] Hong, A. Dragan, and S. Levine. Learning to influence human behavior with offline reinforeement learning, arXiv preprint arXiv:2303. (.XX::2303.02255,, 2023.
[50] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. International conference on machine learning, pages 1889-1897. PMLR, 2015.
[51] F. Huszár. Variational inference using implicit distributions, 2017.
[52] Tien, J. Z.-Y. He, Z. Erickson, Dragan, and S. Brown. Causal confusion and reward
misidentification in preference-basee based reward learning. In The Eleventh International Conference on Learning Representations, 2023.
[53] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and P. Smolley. Least squares generative adversarial networks, 2017.
[54] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two- sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL http: //jmlr.org httgg//pprs/vvl/.1 gt11///rettonla.... html.
[55] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[56] Kostrikov. Pytorch implementations of reinforcement learning algorithms. https: //gi thub. cmm/kkostrikoov/yytrchh-aa http::/pwwoooo-cc--p-------aae-- ack-arltr-gail, -gall 2018

---

[57] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp. Assistive gym: A physics simulation framework for assistive robotics. 2020 IEEE International Conference on Roboties and Aulomation (ICRA), pages 10169-10176. IEFE, 2020.
[58] A. Clegg, Z. Erickson. P. Grady, G. Turk, C. C. Kemp, and C. K. Liu. Learning to collaborate from simulation for assisses dressing. IEEE Robotics and Automation Letters, 5(2): 2746-2753, 2020.
[59] Z. Erickson, Y. Gu, and C. C. Kemp. Assistive VR gym: Interactions with real people to improve virtual assistive robots. CoRR, abs/2007.04959, 2020. URL https:://aaxi..ot http:/iiiv..rgg/////0////0/ 04959.
[60] M. C. Fontaine and S. Nikolaidis, Differentiable quality diversity. CoRR, abs/2206.00488 2021. URL htttps:/aaxxi..rrg/aaa
03894.
[61] J. Mouret and J. Clune. Illuminating search spaces by mapping elites. CoRR, abs/1504.04999, 2015. URL http:://tt//iiv. wotg/br///////// 04909.
[62] Q. Li, Y. Zhai, Y. Ma, and S. Levine. Understanding the complexity gains of single-task rl with curriculum. In International Conference Machine Learning, pages 20412-20455 PMLR, 2023.
[63] A. Gupta, A. Pacchiano, Y. Zhai, S. Kakade, and S. Levine. Unpacking reward shaping: Understanding the benefits of reward engineering sample complexity. Advances in Neural Information Processing Systems, 35:152811 15:15281- 2022.
[64] O. Tunali. Maximum mean discrepancy (mmd) in machine learning, URL.
12

---

Supplementary Material
A
Algorithm
Below provide the pseudo-code for the LARGST-JUMP procedure. Give list of noisily increasing values - in our case, a list of noisily increasing "naturalness" values as we increase Å, the goal of LARGST-JUMP identify the two values, adjacent not, that have the largest gap between them. This enables iterative refinement in Alg. 1.
Algorithm 2 Procedure: LARGEST-JUMP
Require:
1: all A's Aall, all target values S.
2: length of sliding window L
procedure LARGEST-JUMP (S, Aan)
3: Sort S based Aall-
4: lower _earr__ood_______ far, upper upper_ bound_so_faate = []. 0
5: lower_idx_se ii________aa,, upper_ide iid_idx_soo_ sov_
licr__dd_sov___e__e [].
6: high_acc, low_acc = 10, -1
7: high_idx, 1ow low_idx len(S),
8: for = len..,,en((5)) -1 do
9:
if -- then
lower Lover__ouud__o_fft..ppettt lower_idx_so_fale wwpposo......pppeooooooooooo 12: else
10:
11:
istart = max(i I., 0)
14:
Lower_bound_1 far. (mine i+1]))
15:
dower_idx_so..e so_dx_so_aarrrr www.faa..paaaaaaaaaaaaaaaaa append(argmin(S(istaar... i+1])) + istart)
16: end if
17: end for
18: for i- le==ee(S)- do
19: if --i=1 then
20:
upper_bound_ far. (S[1I)
21: upper_idx_so__ so_ app.en (high_idx))
22: else
23:
far ama i+/.))
upper ioper_idx_so___ so far htw.:iicooooffaa.ppppaaaaaaaaaaa appenndaggaaa i+L])) i)
25: end if end for 27: pax_gap_so_ffrr 11, max_gap_idse idxs null
for leielenn S)..., do
29: diff upper_ bew___ooun___ff_ee_
- lower_ boue__bound_soo far
if diff max_gap_so _o__s__faar_ far then
31:
max-gap_so_fal diff
32:
max_gap_idxs
so upper_idx_so__arte
33:
end if
34:
end for
returnAall liox_gap_idxsioo Aai [max_gap_idxs[]]
end procedure
B Additional Details of the Environment Setup
Assistive Gym Itch Scratching We use the itch scratching environment proposed in [57] with the original settings. The biggest difference is that we limit itch positions to randomizing from two fixed points, one in the middle of the forearm and one in the middle of the upper arm, opposed freely sampling from any position on the arms. This is to simplify the robot assistance problem so that we focus on studying robot robustness.

---

Seed
Seed 2
Seed 3
Seed 4
Figure 7: Visualizations of the trajectories of four different co-optimized human-robot policy pairs.
also modify the environment time-span to 100 steps so as to speed up downstream RL training. The reward function in the original itch-scratching environment does not fully capture unwanted behaviors such as the robot swinging its arm and making unwanted contacts. We modify the environment reward function increase the distance penalty (the robot's end-effect being far from the human) and add in contact penalty when the robot impacts areas other than the human's arms.
Co-Ope Co-Optimization We adopt the co-optimization framework proposed in [58], where we have both the human and robot jointly optimize for the task reward. We train both policies using PPO [55]. At every RL. step, we update both the robot and the human policies.
More specifically, we use the ray library and train co-optimized policy pairs using batch sizes of 19,200 timesteps per iteration. We use the default learning rate and PPO hyperparameters in the RLLib library and train for total of 400 iterations.
Training Personalized Robot Policies We keep the human policy from the co-optimized pair as our synthetic human policies. We can then train personalized robot policy to assist the synthetic human. This is akin to programming robotic agents to assist humans. The resulting robot policy which refer to personalized policies the Natural-Adversarial curves. Note that there are different methods to find personalized policies besides running Vanilla RL. We describe them in more detail in Sec. C.
the foeus of the paper and the target which compute
Collecting Canonical Datasets We collect datasets of trajectories synthetic humans and personalized policies as canonical datasets, which we later use train GAN to compute "naturalness", Basically, the GAN enforces that perturbation trajectories stay the proximity the canonical trajectories. Here by "natural", canonical trajectories. This notation can apply to general human robot interaction settings because such applications typically assume canonical trajectories (i.e. default dressing motions), and allow humans to fluctuate within some range of motions.
mean human motions that are indistinguishable from the
Visualizations Here in Fig. 7 we provide visualizations of the trajectories of four different co- optimized human-robot policy pairs. We use the same hyperparameters except for random seeds. The human motions are different amongst different seeds, and remain within reasonable motion range.
C Additional Details of the Experiment Setup
In this section, we talk about the details of different methods for training personalized policies. We also detail the training of GAN for generating aatural-adversarial human behaviors.

---

C.1
Learning Robot Policies
Vanilla RL We use off- te--the-shefff library on PPO algorithm. To facilitate policy training, we use the original robot policy in the co-optimized human-robor pair as the expert to guide the training. More specifically, we query the expert policy for actions and compute Behavior Cloning loss on the robot policy. We set RL loss coefficient to 0.1 and BC loss coefficient 1.
The robot policy is a 4-layer MLP with 100 hidden size. We use batch sizes of 9,600 timesteps per iteration and train for total of 240 iterations. We set environment gamma 0.09. We use learning rate of 0.00005, and an eps of 0.0001. We also set the clip parameter as 0.3 in PPO. In each iteration, we perform 30 epochs of policy optimization, with 20 mini-batch each. We use clipped policy loss, clip gradient norm of robot policy by 20, and clip the value function by 10.
PALM We adopt the same setup as in [6]. To create diverse human distribution, we vary the itch position to randomly sample from anywhere on the two arms. This leads to more diverse human movements. We use recurrent history of 4 timesteps for PALM, and use a 4-layer recurrent VAE with 24 encoder hidden size, and latent size to predict human motions.
Gleeve et al Based on [7]. we diversify the human population from the co-optimization phase. During co-optimization, we jointly train robot policy with 3 human policies initialized from different seeds. The resulting human policies are different from each other and naturally induces diversity in robot training. During the personalization phase, we train robot personalized policy to simultaneously work with the 3 human policies from co-optimization.
Robust GT We perform Robust GT by first computing the manually select adversarial human policies that leads to the lowest robot reward given that the policy naturalness e 0..2, We visualize these failure cases in Sec. F. To train with these adversarial humans, we sample them at 15% rate beside the original synthetic human. While one can train robot policy this way from scratch, we load the previous vanilla RL policy and continue training with this enhanced population. Robust GT can be viewed as a variant of DAgger [42] with simulation-generatted failure cases, or as form of automatic curriculum learning [62, 63).
curve, and then
C.2 Improving GAN training
While GANs are knownly difficult to train, there exists a large number of practical tricks in improving GAN training. We find that the most helpful tricks are: adding noise with annealing. L.SGAN, gradient penalty, and applying different weights for expert and human data. We experimented with training diseriminators concatenated observation and action, and find that it does not lead much change. Thus we end up using observation-only discriminators.
Because human joints move at different rates and scales, it is important to add different amounts of noise to different joint observations. We compute the joint movements from the existing dataset [59]. and multiply each joint movement's standard deviation by factor of 10. We then anneal this by decay rate of 0.98. We apply a gradient penalty of 0.3. We also set the expert loss rate in GAN as 4, and the agent loss rate 1. This helps prevent the discriminator from overfitting to the agent data distribution and collapsing early in the training.
C.3 Computaional Complexity
To generate the adversarial curve in Sec. 5.1, we perform three scans over 1 space, each time launching 18 different RL training in parallel for human policy learning. The three scans can be parallelized, and in total perform 54 policy learning. Within each learning process, we run the PPO algorithm for 120 iterations, 4800 steps per iteration. This results in total of 0.5M environment timesteps. On Intel Xeon Skylake 6130 @ 2.1 GHz CPU, each policy learning 12 CPU core hours. In total, each Natural- Aitural-dveersariall uses 648 CPU core hours.
C.4 Data Requirement
To train the LS-GAN Naturalness Measure, we collect 40 canonical trajectories, each with 100 timesteps, which corresponds to 15 seconds of human-robot interactions. In total, we require 10 minutes of canonical demonstrations.
3

---

C.5 How to Compute Natural-AAdversarial Curve on New Domain?
In order to compute the Natural-Adversarial Curve on new domain, we suggest the following steps:
1. Determine the key features. In the itch-scratching task, we find that rohot poses, velocities and contact forces the key features that determine whether the trajectories dangerous. The naturalness measure is defined by these key features.
2. Determine the size of the canonical dataset. One may steadily increase the size of the dataset, and visualize the Naturalness measure on OOD data as done in Fig. 12. We find that 4,000 timesteps is a reasonable size.
3. Experiment with 2 = 0 to ensure that purely adversarial human policy can be learned. Experiment with A = oa to ensure that natural human policy that stays close to the canonical dataset can be learned. This step ensures that policy learning is working properly.
4. Perform RIGID to scan over 1 and plot the Natural Adversarial Curve.38
D Additional Details of the Natural-Adversaria Frontier
To perform the RIGID algorithm to find the Natural-Adversariis Frontier Curve, we sweep over A E [0.00001, 10]. We perform RIGID algorithm in an iterative refinement manner as in Alg. 1. We keep 3 separate RIGID histories over 3 different random seeds. During each iteration, we select 6 new d's for each seed. We terminate after three iterations. This results total of 54 RL runs per curve.
Plotting in the Intural-Adversamie Coordiate We set the naturalness (x value) of the resulting policies as the mean prediction result from the discriminator. To compute the adversarialness, we normalize the negative robot reward in [200, 1400] range, and clip values that exceed this range to the boundary. We find this range of manually performing different motions VR to find the mean negative reward values of natural motions as well as adversarial human behaviors that lead to failures.
More Natural-Adversarrt Curves We visualize additional Vanilla RL robot policies, trained on differently-seeded synthetic humans.
curves of different
Vanilla RL
Figure 8: Visualizations of more Natural-Adversarre Curves
E
Additional User Study to Verify the Natural-Adversariaa Curve
We perform two additional user studies to verify that (1) the trajectories deemed adversarial in the Curve are indeed dangerous, and that (2) the trajectories deemed natural in the Natural-Adversarial Curve are indeed natural.
To study these hypotheses (1), we sample trajectory pairs from Fig. where they are similar in value (difference < 0.1), but different in y value (difference > 0.3). These correspond to trajectory pairs that are similarly natural, yet exhibit contrasting safety properties. We invite 5 novice users, familiarize them with the canonical trajectories, show them 30 trajectory pairs in randomized orders, and have them answer the following likert scale questions, and compute the final score by using the score of the second question minus the score of the first question. The higher the score is

---

the more dangerous the resulting trajectory is. The result is plotted in the left part of Fig. 9.
Questions for Hypothesis
On a scale from how well do you think the robot achieves the tich-scratching goal?
On scale from to how well you think the robot causes danger?
Different Adversarialness Similar Naturalness
Different Naturalness Similar Adversarialness
Advesurialess
Namralass
-5
-10
Lower Adv
Higher Adv
Lower Nat
Higher Nat
Figure 9: Additional user study results on verifying that (left) under similar naturalness, higher adversarialness in the plot corresponds to trajectories that are more dangerous, and (right) under similar adversarialness, higher naturainess in the plot corresponds trajectories that are more natural
To study these hypotheses (2), we sample trajectory pairs from Fig. 5 where they are similar in y value (difference < 0.1), but different in value (difference 0.3). These correspond to trajectory pairs that are similarly safe, yet exhibit contrasting naturalness. We invite novice users, familiarize them with the canonical trajectories, show them 30 trajectory pairs in random- ized orders, and have them answer the following likert scale question. The higher the score is, the more dangerous the resulting trajectory is. The result is plotted in the left part of Fig. 9.
Questions for Hypothesis 1
On scale from to how close do you think the trajectory resembles the canonical trajectory?
More Visualizations of Robot Failure Cases
In this section, we provide visualizations of the failure cases in both keypoint trajectories and videos. We highlight the human body parts in red to indicate undesirable contact with the robot, such the robot hitting the human's head.
Trajectory Keypoint Visualizations We visualize more failure cases of Vanilla RL as well as robust baselines in Fig. 10.
Video Visualizations We visualize more failure cases of Vanilla RL in videos in Fig. 11.
G Visualizations of the Discriminator
To gain more insight into the Naturalness discriminator, we selectively visualize three different discriminators learned under different Fig. 12. On the left we visualize the discriminators' accuracy on the canonical dataset, with increasing levels of noise added. We find that after applying the tricks in Sec. C.2, the discriminator always able to correctly classify the canonical dataset. As increase the à value, the resulting human trajectories (plotted under "OOD") becomes more

---

Vanilla
PALM
Gleeve et al
Robust GT
Figure 10: Visualization of keypoints of failure trajectories.
adversarial and easier distinguish, which results in higher classification accuracies. On the right we visualize human poses under different noise levels, where the green ones are classified as positive (canonical) and the green ones classified as negative by the discriminator.
H Different Choices of Naturalness Measures
As discussed in Sec. 3, would like to constrain the adversarial policy Aи be similar to the original ди with respect to some f-divergence metric the policy distribution. Commonly used divergence function include x2 divergence, KI. divergence, Wasserstein distance, Such divergence measures are difficult to be estimated and optimized in Eq. (3). Thus, we present the variational form of them. The idea that we can represent them using discriminator optimized with specially designed loss function. The advantage is that the discriminator is differentiable, compatible with Eq. (3), and is provably equivalent the corresponding divergence measures when trained to optimality.
H.1 Proof on Discriminator and x Divergence
Based on [53], we train the discriminator D(T) and the policy Ā(T) using the following loss.
1 Br- ((e))( -
D = min Br-xn [(D(T)
(5)
2
# = arg min Br-zn ()) 1 (
(6)
2
Here a, b are constants, ли is the canonical data distribution of the interaction, learned from human data designed a-priori. Note that though work in the policy learning settings, D and सи are analogous to the discriminator and the generator in the GAN literature. D and सØ are optimized iteratively till convergence. We hereby prove that the resulting मн minimizes the Pearson x divergence. We first derive the optimal discriminator for fixed π in Eq. (5) as:
bAd +x))ar((r) D (')r)
ли (T)

---

0.
04
0.
Figure Visualization videos of failure trajectories. Click on the figure to view the original image, where each trajectory contains a clickable link to its video.
We can then formulate the objective for in Eq. (6) as:
2 = Br-mn (000))-))-BBBxxx(((((1)))))))))1
(T) | By
bun (T) +ax(T) - ?]
E Er-AH
(H(T)+
TH (T)
() dr, set b- 1 and b b- a =2
(T())+
(2 - (T))))
(x(T)) dr
(ΛH x2
Pearson
This means that the optimal Aи optimizes for the x² divergence. practice, we select 1,a 1, == 0 so that
D arg min Br- (D(T)- + B [(exp()(?)+1))) 1)
मи arg min [
(8)
This is the formulation we used in Eq. (2). Q.E.D.
H.2 Proof on Discriminator and KL Divergence
Based on [51, 3), we use the following objective for the discriminator D(r) and the policy #(r):
D arg min B7- (og((+en//DD(t))))EBB..... [egg]I+ep(D((7))
(9)
#(T) = arg arggmnnB..c. [T))
(10)
We can rewrite Eq. (9) as:
D arg min / (eel+xx((D(()))r(( ((oee)))))))( (r)dr
7

---

689
1104
30%
A 0.059499
30%
200
699
595
2 10C0000
705
305
Nose Jome
Figure 12: Visualizations of the discriminator. On left we visualize the discriminators' accuracy on canonical dataset, with increasing levels of noise added. The adversarial human trajectories are plotted under "OOD", On the right we visualize human poses under different noise levels, where the green ones are classified positive (canonical) and the green ones classified negative by the discriminator.
The integral is minimized if and only if the integrand minimized for all r, that Vr, D argmin (1 +cxp(-D(r))) +og ( +e))((((())))((( D
We can then show that the value for (T)) (T) log function in Eq. (10), we get
Jम -7)) Plugging this into the objective
x(>)
Loss(x) = DxL (T)
This means that the optimal AH optimizes for the KL divergence. Q.E.D.
H.3 Discriminator-free Method: Maximum Mean Discrepancy
Maximum Mean Discrepancy (MMD) [54] is a kernel-based statistic test used measure the difference between two distributions, and can be used as loss/cost function in machine learning algorithms for density estimation.
Formally, given random variables X,Y, a feature map & mapping X to another feature space F such that ((X) F. we can the kernel trick to compute the inner product of X, y in F as k(X,Y) = (6(X))$())) We define feature means as a probability measure P on X, which takes ((X) and maps it to the means of every coordinate of d(X):
pr(((x)) = [[ (X)] () IT"
The inner product of feature means of X ~ P and Y ~ Q can be written as ())(((())+HQ(((()) Ep.o E (x)(). (Y)) Maximum Mean Discrepancy is defined
MMD2 (P. = =I|Øp 11иp-~el1
In practice, we can leverage MMD to measure the distance between मम and πи, by collecting datasets of trajectories from each policy, and computing the MMD distance of the two sets. We then

---

plug in MMD to replace y in Eq. (4). This way, the divergence can be directly estimated without training discriminator. We use the MMD implementation in [64], which uses radial basis function as the kernel function.
We then conduct the main experiments using MMD distance the same canonical human-robot policies in Sec. 5 to search for adversarial human policies. Note that we can apply the same RIGID framework automatically scan for values, and compute the aatural-dveersarial frontier. We find that MMD is similarly effective to using LS-GAN as the naturalness metric. For comparison, we take all the resulting adversarial human policies discovered by MMD and plot them under the same LS LS-GAN naturalness plot as in Fig. 5.
1.0
1.0
LS.GAN
MMD
0.8
0.8
Adversaaline messaaalnnssss
Adversarialess
0.6
LS-GAN MMD
8
0.4
0.4
0.2
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Naturaluess
Naturalness
Figure 13: Natural Adversarial human policies found by MMD (blue) plotted alongside the ones found in the main paper (orange).
This shows that while MMD metric underperforms in LS-GAN in certain scenarios in terms of finding adversarial natural human policies, it is also able to find novel scenarios that are not discovered by by LLS-SANN Overall, this suggests that the Ratural-Adversartt framework is applicable different types of naturalness metrics, and using an ensemble of multiple metrics may outperform using a single metric.
I More details on the User Study Setup
We provide more details on the user study in Sec. 5.
L1
VR Visualizations
In the following figure Fig. 14, left and center are the user interacting with virtual robots through the HTC VIVE headset and the hand controller. The right is the first-person view in VR[59].
Figure 14: Workstation for performing the VR user study, where we have the human perform interaction with the robot in the calibrated VR environment.
We first have the users watch 3 iterations of canonical trajectories executed by the personalized robot policies and the original synthetic humans. We then instruct then to perform similar trajectories in their own ways.

---

1.2
Questionaire GUI Interface
For the study on evaluating faithfulness, we use the following interface in Fig. 15, where we display canonical trajectory on the left, and display single-timestep snapshots of two trajectories, one from RIGID policies and one from user VR executions. We ask the user to select which snapshot has stronger correspondence to the left trajectory. We randomize the sequence for each question.
Canonical
B
Figure 15
10


Reviews:
[{"review_id": "OCkV0NooBe", "reviewer": "Reviewer_KeGh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach to assessing the robustness of assistive policies in the presence of humans.  Typical approaches to robustness quantification are adversarial - purely focusing on the worst-case, without considering the realism of the failures.  This paper falls in the line of works seeking to produce robustness for “in-domain” or “in-distribution” failures (here, dealing with abnormal human poses).  The authors propose a solution that visualizes a Pareto frontier over two axes: the degree of “adversarialness” and the degree of “naturalness”, as the output of a learned discriminator the classifies the naturalness of the policy rollout.  Ideally, one would want a small AUC for the curve, meaning that there does not exist natural-behaving trajectories that are adversarial to the robustness of the system.", "summary_of_recommendation": "While the work is in general attacking an interesting problem, I do not feel there are sufficient differences from existing approaches that leverage out-of-distribution (OOD) examples for training, such as variants of DAgger and the like to warrant acceptance.  Moreover, the evaluation and discussion of various aspects, such as the discriminator, computational complexity, and practical utility are lacking.  I do not recommend acceptance as currently written. \n\nUpdate: I have increased my overall score per the revised paper and the authors' rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How can we be assured that the discriminator produces a sensible result for naturalness?\n\nCould the authors please comment on the feasibility of using the approach to train robust policies given the substantial computational overhead required, and any practical limitations (environment complexity, complexity of the safety conditions, etc.)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed approach offers a means for incorporating naturalness into the evaluation criteria of assistive policies.  The authors provide an interesting perspective on the use of naturalness, and the use of models to quantify the performance with respect to how adversarial and natural the rollouts are.  The general principle of using a discriminator seems to be sound, and no concerns in general arise from this choice.  However, this immediately casts the problem as one of learning this discriminator, and so any concerns would be related to the amount of data available to reliably train such a discriminator.\n\nThere are several notable limitations, some of which were pointed out by the authors.  First, human movement was generated from synthetic data generated from a simulator.  Hence, there is still no guarantee that the discriminator can correctly learn a reasonable “naturalness” score.  The authors do not state under what conditions their approach may be applicable nor how generalizable their approach could be, nor discuss any concerns with sim-to-real transfer or how much data is needed from a real human if using human-derived data.  \n\nSecond, the computational overhead of training on the order of 100 policies for evaluation to quantify the AUC seems overly prohibitive and contradicts the premise of practical evaluation of assistive policies.  Moreover, it is not clear how well the trained policies perform against other approaches that purely examine the adversarial evaluation of policies.  It would certainly help to see a comparison of the authors’ method against other approaches that deal only with the adversarial case via ablations and/or comparisons with existing work [9], [10].  It would be good to see both performance and training time reported in such a comparison.\n\nThird, the result is heavily influenced by the discriminator, yet the authors do not evaluate the quality of the trained discriminator, nor do they properly evaluate its classification performance (e.g. using test datasets) and ability to detect out-of-distribution samples.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "HlnR0AxJhIq", "reviewer": "Reviewer_wTf2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In the domain of human motion assistive robots, this paper introduces a natural-adversarial frontier to assess the robustness of a robot's control policy against natural and adversarial human movements. The authors applied RIGID in a simulation environment and confirmed that their proposed method could generate more comprehensive adversarial movements than those created by humans. They also verified that the performance of robust polices can be compared using RIGID.", "summary_of_recommendation": "I believe that quantifying the robustness of the policies of assistive robots is valuable. On the other hand, there is a need to adequately justify the inclusion of a learning algorithm in the quantification.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Descriptions of the relationship between the approach that learns robust policies using adversarial learning and the proposed method\n- Descriptions of the impact of learning instability on the evaluation\n- Detailed descriptions of the experimental settings", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Comments\n\nIn this paper, the authors employ an adversarial learning approach to learning human policy for evaluating the control policies of assistive robots. The objective of this approach is to obstruct the robot's policies with natural movements. In this point, the proposed method is related to robust adversarial reinforcement learning (http://proceedings.mlr.press/v70/pinto17a.html). Therefore, we believe it's necessary to compile related research in this field and clarify its relation to our proposed method.\n\n\nThe method of assessing the robustness of robot control policies inherently incorporates policy learning into the evaluation process. Given that policy learning generally results in unstable performance, it is conceivable that this evaluation metric carries some randomness. What kind of problems might this present when utilizing this method, and how should the authors address these issues?\n\nIn order to efficiently evaluate the strategy using RIGID, I believe it's necessary to properly set the value of lambda. How should it be set for the best experimental results?\n\nThe experimental environment utilized is taken from the environment proposed in [55], hence the description of tasks has been omitted. For efficient comprehension of the experimental content, the basic task setting should be explained within this paper. Moreover, I believe it's crucial to effectively visualize the adversarial and natural movements acquired through adversarial learning. While there are snapshots in the appendix, discerning the naturalness and adversarial nature of the movements is challenging.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "D6n8b6_ihu", "reviewer": "Reviewer_XUqi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper develops robust policies for robots that assist humans. However, this is challenging because measuring robustness is difficult since typical adversarial perturbations may not accurately represent natural human interactions. To address this, the authors propose constructing and analyzing the entire natural-adversarial frontier, representing human policies that balance naturalness and low robot performance. They introduce a method called RIGID, which trains adversarial human policies that trade off between minimizing robot reward and behaving human-like. Using RIGID on an Assistive Gym task, they evaluate the performance of standard collaborative reinforcement learning and existing robustness-increasing methods. The results suggest that RIGID can offer a meaningful measure of robustness, predicting deployment performance, and identifying challenging failure cases in human-robot interaction that are difficult to discover manually.", "summary_of_recommendation": "This paper identified a missing aspect, naturalness, in robust RL, and proposed a viable metric to quantify the performance of a policy's robustness under different naturalness. This aspect is important since in the actual use case of assistive robots, most perturbation from users is unlikely to be unnatural and only a few adversarial users can produce adversarial perturbation. Thus, to my knowledge, this is will be an important evaluation metric in the research of human-robot interaction.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. In line 129, why do you need approximating KL divergence with discriminator?\n2. In Section 5.3, how do you fine-tune vanilla RL?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros\n- **Quality & Clarity**\n  - This paper is well-written and easy to follow. The experiments are well-designed and comprehensive.\n\n\n- **Originality & Significance**\n  - The idea of naturalness-adversarialness frontier is useful, novel, and well-motivated\n  - The proposed adversarial training method that trades off naturalness and adversarialness based on GAIL is well-motivated.\n\nCons\n- The setting of assistive tasks is not explained.\n- The implementation of baselines is not explained (Section 5.3). For example, what is the implementation of RobustGT and PALM? It would be better to explain them briefly\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "AKuUvtldj1B", "reviewer": "Reviewer_KfMg", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a measure to quantify the robustness of a robot policy with respect to natural variations in the human behavior.\nFor measuring the robustness, first a robot policy is trained for given human trajectories. Then, several adversarial human policies are trained, that use GAIL to imitate the human trajectories, but also have an additional reward component to minimize the task reward of the previously learned robot policy. The different GAIL instances use different weights to trade off these two objective, leading to adversarial policies with different levels of naturalness (closeness to the human demonstrations). Robustness is then quantified as the area under the approximated curve that plots negative task reward over naturalness (smaller values indicate higher robustness). In a user study with 9 non-experts, the negated task reward is always below the approximated curve, based on the computed naturalness (using the GAIL discriminator) of the user behavior.", "summary_of_recommendation": "The paper tackles an important problem in HRI. However, I'm currently not convinced about the soundness, reliability and practicability of the proposed measure. I think that it needs to be derived based on the $\\chi^2$ divergence, and Eq.4 should be stated as saddle-point problem. Furthermore, I think that more reliable estimates of the naturalness/divergence should be evaluated. It seems difficult to improve the practicability, but at least the limitations should be more clearly stated (regarding the optimization of the Dec-POMDP that involves a human actor). ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "(-) The data points in the naturalness-adversarialness plots seem to be quite unreliable. When optimizing a policy to minimize a trade-off between task-reward reward and divergence to the human policy, all data points should be on the Pareto frontier (increasing $\\lambda$ should result in higher naturalness and lower adversarial reward). However, based on Fig. 4, some data points are quite far from the frontier. The large noise can be caused both, by learning suboptimal policies, but also by not reliably estimating the divergence for computing the naturalness score. It would be interesting to compare the curves to one where the maximum mean discrepancy to the human demonstration is used for quantifying \"naturalness\". \n\n(-) The Pareto frontier is created based on the samples that achieve largest adversarialness, which introduces a bias due to the noise in the data points, leading to a systematic overestimation of the adversarialness, given a certain level of naturalness. As a result, the results of the user study are not surprising.\n\n(-) Judging from the footnote at page 3, it seems like all human actions and robot actions are sampled at the beginning of the episode and independently from each other. This would be a major limitation that is not well described.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Significance and Originality\n-------------------------\n(+) The submission tackles an important question: How can we quantify robustness with respect to the natural variations within human behavior?\n\n(+) The proposed measure seems novel.\n\n(-) I have concerns regarding the meaningfulness of the computed measure. (see below under quality)\n\n(-) due to high computational costs (multiple runs of reinforcement learning) further questions the usefulness. \n\nClarity\n-------\n(+) The paper is well-written and mostly clear.\n\n(-) However, some important details are not mentioned (See questions).\n\n(-) Eq. 4 seems wrong because the equality does not hold for a fixed discriminator. It needs to be stated as a saddle-point problem ($\\underset{\\pi}{\\max}\\underset{\\mathcal{D}}{\\min}$.\n\n(-) The method is derived based on the reverse KL divergence using the binary cross entropy loss, however, Section 4.1 states that a LS-GAN is used. Why not derive the method based on the $\\chi^2$ divergence?\n\n\nQuality\n-------\n(-) The data points in the naturalness-adversarialness plots seem to be quite unreliable. When optimizing a policy to minimize a trade-off between task-reward reward and divergence to the human policy, all data points should be on the Pareto frontier (increasing $\\lambda$ should result in higher naturalness and lower adversarial reward). However, based on Fig. 4, some data points are quite far from the frontier. The large noise can be caused both, by learning suboptimal policies, but also by not reliably estimating the divergence for computing the naturalness score. It would be interesting to compare the curves to one where the maximum mean discrepancy to the human demonstration is used for quantifying \"naturalness\". \n\n(-) The Pareto frontier is created based on the samples that achieve largest adversarialness, which introduces a bias due to the noise in the data points, leading to a systematic overestimation of the adversarialness, given a certain level of naturalness. As a result, the results of the user study are not surprising.\n\n(-) Judging from the footnote at page 3, it seems like all human actions and robot actions are sampled at the beginning of the episode and independently from each other. This would be a major limitation that is not well described.\n\nMinor Comments\n-----------------------\n- I think (left) and (right) should be swapped in the caption of Fig.1 \n- line 43 \"definite\" should be \"define\"\n\nReferences\n----------\nGretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research, 13(1), 723-773.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an interesting metric for measuring and analysing the robustness and naturalness of HRI policies, by analysing the natural-adversarial frontier. Most reviewers have noted the originality of this work, and the main concerns revolved around the randomnesss of the introduced estimator, and clarifications regarding the presentation of the methodology. The authors have addressed adequately the concerned, and improved both the presentation of their method and their experimental results. After the rebuttal the paper was improved and is an interesting contribution for advancing safe learning in HRI.
Decision: Accept (Poster)