IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors
Authors: Gaurav Datta
Ryan Hoque
Anrui Gu
Eugen Solowjow
Ken Goldberg
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation Learning
Fleet Learning
Energy-Based Models

Abstract: Imitation learning has been applied to a range of robotic tasks, but can struggle when robots encounter edge cases that are not represented in the training data (i.e., distribution shift). Interactive fleet learning (IFL) mitigates distribution shift by allowing robots to access remote human supervisors during task execution and learn from them over time, but different supervisors may demonstrate the task in different ways. Recent work proposes Implicit Behavior Cloning (IBC), which is able to represent multimodal demonstrations using energy-based models (EBMs). In this work, we propose Implicit Interactive Fleet Learning (IIFL), an algorithm that builds on IBC for interactive imitation learning from multiple heterogeneous human supervisors. A key insight in IIFL is a novel approach for uncertainty quantification in EBMs using Jeffreys divergence. While IIFL is more computationally expensive than explicit methods, results suggest that IIFL achieves a 2.8x higher success rate in simulation experiments and a 4.5x higher return on human effort in a physical block pushing task over (Explicit) IFL, IBC, and other baselines.

IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors
Gaurav Datta 11 Ryan Hoque" Anrui Gu', Eugen Solowjow?, Ken Goldberg!
AUTOLab at UC Berkeley Siemens Research Lab
Imitation learning has been applied to range of robotic tasks, but can struggle when robots encounter edge cases that are not represented in the training data (i.e., distribution shift). Interactive fleet learning (IFL.) mitigates distribution shift by allowing robots access remote human supervisors during task execu- tion and learn from them over time, but different supervisors may demonstrate the task in different ways. Recent work proposes Implicit Behavior Cloning (IBC), which is able to represent multimodal demonstrations using energy- energy-based mod- els (EBMs). In this work, we propose Implicit Interactive Fleet Learning (IIFL), an algorithm that builds on IBC for interactive imitation learning from multiple heterogeneous human supervisors. A key insight in IIFL. novel approach for uncertainty quantification in EBMs using Jeffrees divergence. While IIFL is more computationally expensive than explicit methods, results suggest that IIFI achieves 2.8x higher success rate in simulation experiments and 4.5x higher return on human effort in physical block pushing task over (Explicit) IFL, IBC, and other baselines.
Abstract:
Introduction
Imitation learning (IL), the paradigm of learning from human demonstrations and feedback, has been applied to diverse tasks such as autonomous driving [1, 2, 31. robot-assisted surgery [4, 5], and deformable object manipulation [6, 7, 8). The most common II. algorithm is behavior cloning (BC) [2], where the robot policy is derived via supervised machine learning on an offline set of human task demonstrations. Since BC can suffer from distribution shift between the states visited by the human and those visited by the robot, interactive IL (IIL) algorithms including DAgger [9] and variants [10, 11, 12] iteratively improve the robot policy with corrective human interventions during robot task execution. These algorithms are typically designed for the single-robot, single-humat setting; interactive fleet learning (IFI.) [13] extends III. multiple robots and multiple human supervisors. However, learning from multiple humans can be unreliable the data is often multimodal.
Training data is multimodal when the same state is paired with multiple (correct) action labels: {(s, a.). (s, ay,ay).... }, # aj. Almost all robot tasks such as grasping, navigation, motion plan- ning, and manipulation can be performed in multiple equally correct ways; as result, almost all demonstration data has some degree of multimodality. Multimodality is especially severe when learning from different human supervisors with varying preferences and proficiency, they demon- strate the same task in different ways [14]. Multimodality can also occur in the demonstrations of one individual human who may make mistakes, become more proficient at the task over time, or execute a different valid action when subsequently encountering the same state [14, 15).
Florence et al. [16] propose Implicit Behavior Cloning (IBC), an algorithm that trains an energy- based model (EBM) [17] to represent state-action mappings implicitly rather than explicitly. While this makes model training and inference more computationally expensive (Section 6), implicit mod-
Equal contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

BC
IBC
1IFL
(A) Robot Paths
Junction
Hallway
Wind
(B) IIL Energy
Figure 1: In the 2D navigation experiments from Section 5.1, the robot must navigate from the blue X marker the left the green marker the right, where the robot go either above below the rectangular grey obstacle and continue through section subject upward wind forces (blue arrows) that shift commanded motions upward. (A) Robot Trajectories: After training on 100 demonstrations
the two paths around the obstacle, pure behavior cloning cannot make progress past the fork due to multimodal demonstrations, while Implicit Behavior Cloning cannot overcome the distribution shift due to wind in the +y direction execution time (denoted in light blue). IIFL reaches the goal by handling both multimodality and distribution shift. (B) Implicit Interactive Fleet Learning Energy: We display normalized IIFL energy distributions from representative states in the trajectory. Lower energy (darker) indicates more optimal action, and the x and axes are the 2D action deltas that the robot can execute (which can be mapped directly onto the corresponding 1x I cell in the maze). At the junction point, both upward and downward actions attain low energy; in straight hallway, the rightmost actions attain low energy; (making progress toward the goal while fighting the wind) attain low energy.
windy area, actions toward the lower right comnr
els can represent multiple actions for each state. This property allows them to handle both single- human multimodality and multi-human heterogeneity, as they are indistinguishable from a data- centric perspective. However, IBC suffers from the same distribution shift as (Explicit) BC.
In this paper we combine implicit models with interactive fleet learning to facilitate interactive learn- ing from multiple humans. See Figure for intuition. As existing IFL algorithms rely on estimates of epistemic uncertainty like the output variance among an ensemble of networks, which are incom- patible with implicit models (Section 4.3), we propose a new technique for estimating the epistemic uncertainty in EBMs using Jeffreys divergence [18].
This paper makes the following contributions: (1) Implicit Interactive Fleet Learning (IIFL), the first IILL algorithm to implicit policies, (2) novel metric for estimating uncertainty in energy mnesyy-based models, (3) simulation experiments with a fleet of 100 robots and 10 heterogeneous algorithmic su- pervisors, (4) physical experiments with fleet of 4 robots and 2 heterogencous human supervisors. Open-source Python code available at https: httgg/gaa.c///rrereelllyatttttoa I
thub. com httr/BBerkellllye
2 Preliminaries and Related Work
2.1 Interactive Imitation Learning
Learning from offline set of human task demonstrations with behavior cloning (i.e., supervised
learning) is an intuitive and effective way to train robot control policy [19, 2, 6, 1]. However, behavior cloning can suffer from distribution shift [9], as compounding approximation errors and real-world data distributions (e.g., variable lighting in a warehouse) can lead the robot to visit states that were not visited by the human. To mitigate distribution shift, Ross et al. [9] propose dataset aggregation (DAgger), an III. algorithm which collects online action labels on states visited by the robot during task execution and iteratively improves the robot policy. Since DAgger can request excessive queries to human supervisor, several IIL algorithms seek to reduce human burden by intermittently ceding control to the human during robot execution based on some switching criteria

---

[11, 10, 20]. Human-gatee IIL [11, 21, 22] has the human decide when to take and cede control, while robot-gated II [23, 10, 12, 20J has the robot autonomously decide. Hoque et al. [13] propose Interactive Fleet Learning (IFL), which generalizes robot-gated III. to multiple robots supervised by multiple humans. In this work, we consider the IFL setting.
Sun al. [24] propose method for interactive imitation learning from heterogeneous experts, but their method is not based on implicit policies and is limited to autonomous driving applications. Gandhi et al. [25] also interactively learn from multiple experts and propose actively soliciting the human supervisors to provide demonstrations that are compatible with the current data. However, this prevents the robot from learning alternative modes and requires the human supervisors to comply with suggestions, which may not occur due to human suboptimality, fatigue, or obstinacy [26].
2.2 Robot Learning from Multimodal Data
Learning from multimodal demonstrations an active challenge in machine learning and robotics. A mixture density network [27] is popular approach that fits a (typically Gaussian) mixture model the data, but requires setting a parameter for how many modes fit, which may not be known priori. When actions can be represented as pixels in image (e.g.. pick points), Fully Convolu- tional Network [28] can be applied to learning pixelwise multimodality [8, 29]. Shafiullah al. [30] propose Behavior Transformers, technique that applies the multi-token prediction of Transformer neural networks [31] to imitation learning. Other Transformer based policies report similar benefits for multimodal data [32, 33]; however, these approaches require action discretization to cast behav- ior prediction as next-token prediction. a very recent paper, Chi et al. [34] introduce diffusion policies, an application diffusion models [35] to imitation learning from multimodal data.
Florence et al. [16] propose implicit behavior cloning, technique that trains conditional energy- based model [17] and is found to outperform (explicit) BC and mixture density networks in their experiments. As opposed to explicit models that take the form æ S + A. implicit models take the form of a scalar-valued function F. S x A + R; the action is an input rather than an output of the model. To sample an action from the policy, instead of evaluating the explicit model T(s), the implicit model must perform optimization over E conditioned on state st
à arg min E(s, a)
aad
In this work, we combine IBC with IFI. to mitigate the effects of both distribution shift and multi- modality. To our knowledge, we are the first to extend implicit policies to interactive IL.
2.3 Jeffreys Divergence
The Jeffreys divergence [18] is a statistical measure of the distance between two probability distri- butions and is a symmetric version of the Kullback-Leeib (KI.) divergence:
D)())) = Dz.. (KI(P)P).
The KL divergence is widely used in machine learning algorithms, most commonly in variational autoencoders [36] and generative adversarial networks [37]. It has also been used for dimensionality reduction [38], information bottlenecks [39], and policy gradient methods for reinforcement learning [40, 41]. The Jensen-Shannon divergence [42] is another symmetric KL divergence that sums the KL divergences of both distributions against the mixture of the two, but neither the Sensen-Shannon nor the asymmetric KL divergences have the structural properties that make Jeffreys divergence amenable to our setting (Section 4.3). Nielsen [43] derives a proposition similar to Identity 1 (Sec- tion 4.3) with Jeffreys divergence for exponential families but does not apply to energy-based models. To our knowledge, IIFL is the first algorithm to Jeffreys divergence for uncertainty estimation in energy- based models, exploiting its structural properties for fast computation.
3

---

Problem Statement
We consider the interactive fleet learning (IFL) setting proposed by Hoque et al. [13]. fleet of N robots operate in parallel independent Markov Decision Processes (MDPs) that are identical apart from their initial state distributions. The robots can query set of M NV human supervisors with action space Ah = AU (R), where E AA is teleoperation in the action space of the robots and R is "hard reset" that physically resets robot in failure state (e.g., delivery robot tipped over on its side). As in [13], we assume that (1) the robots share policy To, S + A, (2) the MDP timesteps are synchronous across robots, and (3) each human can only help one robot at a time. However, unlike the original IFL formulation [13], we do not assume that the human supervisors are homogeneous; instead, each human may have a unique policy Tij S - Arr. Furthermore, each T may itself be nondeterministic and multimodal, but assumed be optimal or nearly optimal.
An IFL supervisor allocation algorithm is a policy that determines the assignment a' of humans to robots at time t, with no more than one human per robot and one robot per human at time:
w (s', no.))) + a' E 10.1)XxM s.t.
Σ a S and
ogy S
Vi,j.
The allocation policy in IFL. must be autonomously determined with robot-gated criteria [10, 12] rather than human-gatee criteria [11, 21, 22] in order to scale to large ratios of N to M. The IFL objective is to find an w that maximizes return on human effort (ROHE), defined as the average performance of the robot fleet normalized by the amount of human effort required [13]:
M N
Fis
max E P.rpooo WED
(T)
E ww(s To (x'-1x'1) x
where || || is the Frobenius norm, T is the amount of time the fleet operates (rather than an individual episode horizon), and 0g are the initial parameters of ne.
4 Approach
4.1 Preliminaries: Implicit Models
We build on Implicit Behavior Cloning [16]. IBC seeks to learn a conditional energy-based model E: S X A + R, where E(s,, a) is the scalar "energy" for action a conditioned on state s. Lower en- ergy indicates a higher correspondence between & and a. The energy function defines a multimodal probability distribution π of action conditioned on state s:
((E,,)))
π (a|s)
(4)
7(s)
where 7(s) normalization factor known as the "partition function." In practice, we estimate E with learned neural network function approximator E, parameterized by and train Eg on samples {ss, ai} collected from the expert policies ии. Training Eg is described in Appendix 7.2.
4.2 Implicit Interactive Dataset Aggregation
Behavior cloning is prone to distribution shift due to compounding approximation errors [9]. and any data-driven robot policy may encounter edge cases during execution that are not represented in the training data [13]. We extend IBC to interactive imitation learning using dataset aggregation of online human data, and iteratively update the shared robot policy with the aggregate dataset at fixed interval 1 < / T via supervised learning, in DAgger [9] and variants [11, 13]:
D + D' D D'D where Dú = {(}), 7 (s:)) The (st) + and Σ/11 at = =1}
no, <- ming L(TO, Dd), if f = 0 (mod 1)

---

where TH (s{) is the teleoperation action from human j for robot i at time t, and agj is the assignment of human j to robot i at time 1, as in Equation 2. Ross et al. [9] show that such a policy incurs approximation error that is linear in the time horizon rather than quadratic, as in behavior cloning.
4.3 Uncertainty Estimation for EBMs
While prior work computes the output variance among a bootstrapped ensemble of neural networks estimate epistemic uncertainty [44, 12, 10], this approach is not applicable to implicit policies because multimodality results in false positive: different ensemble members may select equally optimal actions from different modes, resulting in high variance despite high certainty, Furthermore, training and inference in EBMs are much more computationally expensive than in explicit models (Section 6), making ensembles of 5+ models impractical. Finally, inference in implicit models is nondeterministic, creating an additional source of variance that is not due to uncertainty.
The notion of ensemble disagreement can still be applied to EBMs by considering the action distri- butions at given state rather than the single predicted actions. At states within the distribution of the human data, bootstrapped EBM will predict action distributions that are concentrated around the human actions. However, outside of the human data distribution, the models have no reference behavior to imitate, and will likely predict different conditional action distributions due to random initialization, stochastic optimization, and bootstrapping. Accordingly, we propose bootstrapping implicit policies and calculating the Jeffreys divergence D, [18] between them as a measure of how their conditional action distributions differ at given state. Jeffireys divergence in this setting has two key properties: (1) it is symmetric, which is useful neither bootstrapped policy is more correct than the other, and (2) it is computationally tractable for EBMs as it does not require estimating the partition function (s) (Equation 4). To show (2), we derive the following novel identity (proof in Appendix 7.1):
Identity 1. Let E₁ and Eg be two energy-based models that respectively define distributions π and T2 according to Equation 4. Then,
D, (Ti (=15)][(2(2(11)) (aaon (2 a) Fy (s, (ewrs((1))) (F) E2(8, a))
Crucially, the intractable partition functions do not appear in the expression due to the symmetry of Jeffreys divergence. We estimate the expectations in Identity using Langevin sampling. Note that this method is not limited to the interactive IL setting and may have broad applications for any algorithms or systems that use energy-based models. We provide more intuition on the proposed metric in Figure the appendix and consider how this method may be generalized to a greater number models in Appendix 7.3.
4.4 Energy-Based Allocation
To extend IBC to the IFL setting, we synthesize the Jeffreys uncertainty estimate with Fleet-DAgger [13]. Specifically, we the Fleet-DAgger priority function (s, Te,) - [0, 2)) to prioritize robots with high uncertainty, followed by robots that require hard reset R. This produces super- visor allocation policy w with Fleet- Freer-EnsembllDAgeer the U.C. Cnncertainty-Constraiii allocation scheme in [13]. We refer to the composite approach as IIFL.
5
Experiments
5.1 Simulation Experiments: 2D Navigation
To evaluate the correctness of our implementation and provide visual intuition, we first run experi- ments in a 2D pointbot navigation environment. See Figure 1 for the maze environment, representa- tive trajectories, and energy distribution plots. We consider discrete 2D states s (I, y) N22 (the Cartesian pose of the robot) and continuous 2D actions (Az, Ay) [-1, (relative changes in Cartesian pose). The maze has fixed start and goal location and consists of forked path around
5

---

Cumulative Successes
Cumulative Hard Resets
Return on Human Effort
Ball Balance
Ant
Anymal
BC
IBC
IIFL-
IIFL
Figure 2: IFL Benchmark simulation experiment results. Despite unimodal supervision, IIFL is competitive with or outperforms IFL and other baselines across 3 environments, suggesting benefits of implicit policies beyond robustness multimodality, Shading represents #1 standard deviation.
large obstacle followed by long corridor. An algorithmic supervisor provides 100 demonstrations of the task, randomly choosing go upward downward at the fork with equal probability. Since a model can simply overfit to the demonstrations in this fow-dimensional environment, to induce distribution shift we add "wind" at execution time to a segment the right corridor with magnitude 0.75 in the +y direction.
In 100 trials, (explicit) BC achieves a 0% success rate, IBC achieves 0% success rate, and IIFL achieves 100% autonomous success rate (i.c., robot-only trajectories without human interventions, after interactive training). In Figure 1 we observe that BC cannot pass the fork due to averaging the two modes to zero. Meanwhile, IBC is not robust to the distribution shift: once the wind pushes the robot to the top of the corridor, does not know how to return to the center. We also observe that the IIFL energy distributions in Figure I(B) reflect the desired behavior accordance with intuition.
5.2 Simulation Experiments: IFL Benchmark
Environments: Evaluating IIFL in simulation is uniquely challenging as it requires the fol- lowing, precluding the use of most existing benchmarks in similar papers: (1) efficient simulation of large robot fleets, (2) simulation of multiple algorithmic humans, (3) interactive human control, and (4) heterogeneous human control, which is difficult specify in joint space. To accommodate these requirements, following prior work [13] we evaluate with Isaac Gym [45] and the IFL Bench- mark [13]. We separate these experiments into two domains: (1) homogeneous human control in 3 environments (Ball Balance, Ant, Anymal) to compare with prior IFL. algorithms that assume unimodal supervision: (2) heterogeneous human control in FrankaCubeStack, the only Isaac Gym environment with Cartesian space control. More details are available in Appendix 7.4.
Following prior work [13], measure the total successful task completions across the fleet and the total number of hard resets. For interactive algorithms, we also measure the return on human effort (Equation 3) where reward a sparse E {0, }} for task completion. Task execution deemed successful if the robot completes its trajectory without a hard reset and reaches 95% of expert human reward.
Metrics:
Baselines: We compare IIFL the following baselines: (explicit) BC, IBC, (explicit) IFL (specif- ically, Fieet-EnsembleDAgger [13]), and IIFL-Random (IIFL-)), which is ablation of IIFL that

---

Algorithm
Avg. Reward
Task Successes
ROHE
BC
29.27 + 14.05
0.3 0.3:0.05
N/A
IBC
24.96 + 0.83
0.0
N/A
IFL
230.39 = 53.41
2.2
2.30 + 0.53
IIFL-
166.24 = 28.63
0.0
1.66 + 0.29
IIFL
784.26 + 122.41
26.7 + 4.5
7.84 1.22
Table I: Exccution results from the FrankaCubeStack Isaac Gym environment with heterogeneous expert
policies. IIFL significantly outperforms the baselines average reward, task successes, and return on human effort.
allocates humans to robots randomly instead of using the Jeffreys uncertainty estimate. Human supervisors for BC and IBC perform only hard resets (i.e., no teleoperation) during execution.
Experimental Setup: We run experiments with a fleet of N = 100 robots and M = 10 algorith- mie supervisors, where the supervisors are reinforcement learning agents trained with Isaac Gym's reference implementation of PPO [41]. All training runs have hard reset time tg timesteps, minimum intervention time ty 5 timesteps, and fleet operation time T 10, 000 timesteps [13], and are averaged over 3 random seeds. The initial robot policy Too for all algorithms is initialized with behavior cloning on 10 full task demonstrations. While IFI. trains at every timestep follow- ing prior work [13], the implicit interactive algorithms train at intervals of 1000 timesteps with an equivalent total amount of gradient steps for increased stability of EBM training.
Stack, in which Franka arm grasps a cube and stacks it on another (see Appendix 7.4.2 for images and details), has several differences from the other 3 environments. First, since it allows Cartesian space control, we can script heterogeneous supervisor policies with grasps corresponding to each face of the cube; the M = 10 supervisors are split into 4 groups, each of which has unique policy. Second, due to the difficulty scripting interactive experts, the online interventions take place at execution-ting (i.e., the robot policy is frozen). Third, since there is no notion of catastrophic failure in the cube stacking environment, we do not report hard resets as there are none.
The results are shown in Figure 2 and Table 1. In the homogeneous control experiments, we observe that IIFL rivals or outperforms all baselines across all metrics, with the exception of hard resets in the Anymal environment. We hypothesize that the latter results from learning more "aggres- sive" locomotion that makes greater progress on average but is more prone to failure. These results suggest that implicit policies may have desirable properties over explicit policies such as improved data efficiency and generalization even when multimodality not present in the data, as suggested by prior work [16]. The severity of distribution shift due compounding approximation error [9] in the homogeneous experiments roughly corresponds to the performance gap between BC and IFL. (or IBC and IIF.). Surprisingly, (explicit) IFL underperforms BC in Ball Balance; we hypothesize that this is due to its frequent policy updates on a shifting -ow-dimensional data distribution. In the FrankaCubeStack environment, IIFL significantly outperforms the baselines across all metrics, indicating the value of implicit policies for heterogeneous supervision. The 74% performance gap between IFL. and IIFI. corresponds to the severity of multimodality in this environment. Only IFI. and IIFL attain nontrivial success rates; while IIFL-R makes progress, it is not able successfully stack the cube, suggesting that IIFL allocates human attention more judiciously.
Results:
5.3 Physical Experiments: Pushing Block to Target Point amid Obstacle
Experimental Setup: To evaluate IIFL in experiments with real-world human multimodality and high-dimensional state spaces, we run an image-based block-pushing task with fleet of N= 4 ABB YuMi robot arms operating simultancously and M = 2 human supervisors, similar to Hoque et al. [13]. See Figure 3 for the physical setup. Each robot has an identical square workspace with a small blue cube and rectangular pusher as an end effector. Unlike Hoque et al. [13], we add a square obstacle to the center of each workspace. The task for each robot to push the cube to a goal region diametrically opposite the cube's initial position without colliding with the walls or the obstacle. Once this is achieved, the goal region is procedurally reset based the new cube position. As described in Section 3, the role of human superivsion is to (1) telcoperate when requested and (2)
7

---

YuMi #1
YuMi #2
Observation
2
3
4
Actions
Figure 3:
Physical experiment setup with 2 ABB YuMi robots for a total of 4 independent arms.
Algorithm
Successes (1)
Hard Resets (1)
ROHE (t)
BC
2.0 0.8
51.0 + 0.8
N/A
IBC
20.3 4.1
35.3 + 6.8
N/A
IFL.
7.0 0.8
47.3 + 0.5
0.13 + 0.01
IIFL
36.3
3.2
0.71
Table 2: Physical block pushing experiment results. IIFL outperforms all baselines number of task successes and ROHE and explicit methods in hard resets. Implicit BC and IIFL incur similar amounts of hard resets.
provide physical hard reset when requested. When both paths to the goal are equidistant, Human I pushes the cube clockwise around the obstacle while Human 2 pushes the cube counterclockwise; if one path is closer, the human takes that path. Hard resets R are defined to be collisions of the cube with the obstacle or the boundaries of the workspace. Furthermore, unlike the discrete action space in Hoque et al. [13], we use a continuous 2D action space of a = (Ax, Ay) that corresponds the vector along which to push the block, starting from the block's 's center. We run 3 trials of each algorithm in Table for T 150 timesteps; see Appendix 7.4.3 for
details.
The results are shown in Table 2. We observe that implicit policies are crucial for success, as the explicit methods rarely reach the goal and incur many hard resets, Results suggest that IIFL improves the success rate by 80% over IBC and improves ROHE by 4.5x over IFL. However, IIFL incurs similar number of hard resets to IBC. We hypothesize that the duration of the physical experiment, difficult to extend due the significant robot and human time required, is insufficient to learn subtle collision avoidance behaviors that noticeably reduce the number of hard resets.
Results:
6 Limitations and Future Work
Since IIFL extends IBC, inherits some of its limitations. First, model training and inference require 18x and 82x more computation time than explicit methods: one NVIDIA V100 GPU, we measure implicit training to take an additional 0.34 seconds per gradient step and implicit inference to take an additional 0.49 seconds. Second, Florence et al. [16] find that IBC performance falls on some tasks when the action space dimensionality very high (AA) > 16); we do not observe this in our experiments as |A] < 12 but IIFL likely incurs this property with higher-dimensional actions. Third, while is 7x faster than alternate methods for implicit models and has sub-second latency for a fleet of 100 robots, IIFI uncertainty estimation is nevertheless 340x slower than its highly efficient explicit counterpart (Appendix 7.4.4). Finally, the real-world evaluation of IIFL is limited to block pushing with fixed block properties; more comprehensive evaluation of IIFL in wider range of physical domains is required to assess its full applicability.
In future work, we will evaluate IIFL in additional physical environments as well extend recently proposed alternative approaches for handling multimodality such as Behavior Transformers [30] and Diffusion Policies [34] the IFL setting. We will also develop algorithms that effectively learn from human demonstrations that are not only multimodal but also suboptimal. We note that as the Jeffreys uncertainty quantification method does not rely on any IFL assumptions, it may be broadly useful beyond this setting to any applications involving Boltzmann distributions and EBMs.

---

Acknowledgments
This research was performed the AUTOLab UC Berkeley in affiliation with the Berkeley Al Research (BAIR) Lab. The authors were supported in part by donations from Siemens, Google, Bosch, Toyota Research Institute, and Autodesk and by equipment grants from PhotoNeo, NVidia, and Intuitive Surgical. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. We thank our colleagues who provided helpful feedback, code, and suggestions, especially Cesar Salcedo, Letian Fu, and Aviv Adler.
References
[1] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots. Agile au- tonomous driving using end-to-end deep imitation learning. Robotics: Science and Systems (RSS), 2018.
[2] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touret- zky, editor, Neural Information Processing Systems (NeurIPS), volume 1. Morgan-Kaufmann 1988.
[3] J. Chen, B. Yuan, and M. Tomizuka. Deep imitation learning for autonomous driving in generic urban scenarios with enhanced safety. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2884-2890, 2019.
[4] Paradis, M. Hwang, Thananjeyan, J. Ichnowski, D. Seita, D. Fer, Low, J. E. Gonzalez, and K. Goldberg. Intermittent visual servoing: Efficiently learning policies robust to instrument changes for high-precision surgical manipulation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 7166-7173, 2021.
[5] W. Kim. P. Zhang. P. Gehlbach, Iordachita, and M. Kobilarov. Towards autonomous eye surgery by combining deep imitation learning with optimal control. In Conference Robot Learning (CoRL), 2020.
[6] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, et Deep imitation learning of sequential fabric smoothing from algorithmic supervisor. In JEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 9651-9658, 2020.
[7] Y. Avigal, L. Berscheid, Asfour, T. Kroger, and K. Goldberg. Speedfolding: Learning efficient bimanual folding garments. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-8, 2022.
[8] R. Hoque, K. Shivakumar, S. Aeron, G. Deza, A. Ganapathi, A. Wong, J. Lee, A. Zeng, V. Van- houcke, and K. Goldberg. Learning to fold real garments with one arm: case study in cloud-based robotics research. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[9] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, 2011.
[10] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. ThriftyDAgger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning (CoRL), 2021.
[11] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Au- tomation (ICRA), pages 8077-8083, 2018.

---

[12] K. Menda, K. proach to Safe Imitation Learning. In /EEE/RS) International Conference on Intelligent Robots and Systems (IROS), 2019.
and M. J. Kochenderfer. EnsembleDAgger: A Bayesian Ap-
[13] R. Hoque, L. Y. Chen, Sharma, K. Dharmarajan, Thananjeyan, P. Abbeel, and K. Gold- berg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Con- ference on Robot Learning (CoRI.), 2022.
[14] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart in-Mart in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[15] C. G. Northcutt, A. Athalye, and J. Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Neural Information Processing Systems (NeurIPS), 2021.
[16] R. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning (CoRL), 2021.
[17] Y. LeCun, S. Chopra, R. Hadsell, A. Ranzato, and F, J. Huang. A tutorial energy-based learning. Predicting Structured Data, 1(0), 2006.
[18] H. Jeffreys. The Theory of Probability. Oxford University Press, 1939.
[19] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-488) 2009.
[20] J. Zhang and K. Cho. Query-efficient imitation learning for end-to-end autonomous driving.
In Association for the Advancement of Artificial Intelligence (AAAI), 2017.
[21] Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. Science and Systems (RSS), 2020.
In Robotics:
[22] H. Liu, S. Nasiriany, L.. Zhang, Z. Bao, and Y. Zhu. Robot learning on the job: Human-in-the- loop autonomy and learning during deployment. arXiv, 00bs/2211.084111 2022.
[23] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. Novoseller, and K. Goldberg. LazyDAgger: Reducing context switching in interactive imitation learning. In IEEE Conference on Automation Science and Engineering (CASE), pages 502-509, 2021.
[24] X. Sun, S. Yang, and R. Mangharam. Mega-dagger: Imitation learning with multiple imperfect experts. ArXiv, arXiv preprint arXiv:2303.00633, 2023.
[25] K. Gandhi, S. Karamcheti, M. Liao, and D. Sadigh. Eliciting compatible demonstrations for multi-human imitation learning. In Conference on Robot Learning (CoRL), 2022.
[26] S. E. F. Chipman. The Oxford Handbook of Cognitive Science. Oxford University Press, 10 2017. ISBN 9780199842199
[27] C. M. Bishop. Mixture density networks. Neural Computing Research Group Report, 1994.
[28] E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks for semantic segmentation.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:640-65 651, 2017.
[29] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRL), 2020.
10

---

[30] N. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. In Neural Information Processing Systems (NeurlPS). 2022.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Neural Information P Processing Systems (NeurIPS), 2017.
[32] M. Shridhar, I.. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning (CoRL), 2022.
[33] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. VIMA: General robot manipulation with multimodal prompts. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
[34] C. Chi, S. Feng. Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0413 2023.
[35] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. / 1239, 2020.
[36] D. P. Kingma and M. Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
[37] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Advances in Neural Information Process- ing Systems, 2014.
(38) I.. van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86)25599-205,, 2008 URL http://jmlr http://mmmrrrr.peeeeeeeeeeee patgpappers//////// vandermaaten08a. html.
[39] N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1 1-5, 2015.
[40] J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimiza- tion. In International Conference on Machine Learning, 2015.
[41] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint wtXiv:1707.06347 2017.
[42] Lin. Divergence measures based on the shannon entropy. IEEE Trunsactions on Information Theory, 47(1)14 1991.
[43] F. Nielsen. Fast approximations of the jeffreys divergence between univariate gaussian mix- tures via mixture conversions to
distributions. Entropy, 23, 2021.
[44] K. Chua, R. Calandra, R. T. McAllister, and S. Levine. Deep reinforcement learning in a hand- ful of trials using probabilistic dynamics models. In Newral Information Processing Systems, 2018.
[45] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108, 10470, 2021.
[46] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. ArXiv preprint / 2018
[47] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, (CML'11, page 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 7781450306195.
[48] R. M. Neal. Annealed importance sampling. Statistics and computing, 11:125-133 2001.

---

7 Appendix
7.1 Jeffreys Divergence Identity
We derive the following identity from the main text:
Identity 1. Let Ei and En be two energy-based models that respectively define distributions T and
22 according to Equation Then,
D, (Λ
Eawa (-1a) [g a) (s, a)] + (E)(s,)) E2 a)l
Proof. The proof follows from applying the definition of Jeffreys divergence EBMs:
D, (Ta (m1((1s)] DxL (N1 (-1s)|| + DKL (12
(-))(())]] (-1s))
log Ti (als)
(als)
E
T2(aas}] (a
log 1i (a|s)
Eawa, (-1s) [8) - En (8,a)] - log (8) + log (2(8)
+ (x-=x((1s) (s, E2(s, log (2 + log Zi(s)
Ea (11) (g(s, a) Ei + E (E](s, - E2(s,
To provide more intuition on this identity, we plot the Jeffreys divergence for pair of isotropic Gaussian energy functions in Figure 4.
7.2 Additional Details on Implicit Models
Implicit BC trains an energy-based model Eg on samples {ss, as collected from the expert poli- cies иH. After generating a set of counter-examples {ā?} for each 8,, Implicit BC minimizes the following InfoNCE [46] loss function:
(-Elssia))
L Σ
log (a()), po(a,|s; (ā(1) e-Eo(s,,a)) ES Fo
(5)
This loss is equivalent to the negative log likelihood of the training data, where the partition func- tion Z(s) is estimated with the counter-examples. Florence et al. [16] propose three techniques for generating these counter-examples { and performing inference over the learned model Eg; we choose gradient-based Langevin sampling [47] with an additional gradient penalty loss for train- ing in this work as Florence et al. [16] demonstrate that it scales with action dimensionality better than the alternate methods. This a Markov Chain Monte Carlo (MCMC) method with stochastic gradient Langevin dynamics. More details are available in Appendix B.3 of Florence al. [16].
We use the following hyperparameters for implicit model training and inference:
Hyperparameter
Value
learning rate learning rate decay
0.0005
0.99
learning rate decay steps
100
train counter examples
8
langevin iterations
100
langevin learning rate init. langevin learning rate final
0.1
le-5
langevin polynomial decay power
2
inference counter examples
512
Table 3: Implicit model hyperparameters.
12

---

Energy Functions
Jeffreys Divergence
0.010
-100,02
0.005
-100.44
0.84
2.
100.66
-.05
60020
1000.8
9.0
0.015
4.8
100.22
-0.12
0,000
-100.14
-011
E
4.0 -32
244
-166
6.8
State
0.
0.8
16
2.4
4.0
State
Figure 4: Consider pair of isotropic Gaussian energy functions (i a) and Ez(s, a) in green and purple respectively, where each function is negated Gaussian probability density function and E₁ adds uniform offset of Z - 100 all values (Left). Using numerical integration to directly compute the expectations in the Jeffreys divergence identity (Identity 1), each state calculate the distance hetween the implicit policies defined by two energy functions (Right). intuition suggests, the divergence peaks the mean of each Gaussian (where one energy function is highest and the other is near zero) and approaches zero where the energy functions are the same (at the center and edges of the state space). Note the symmetric structure of the Jeffreys curve, which produces identical values regardless of the offset Z.
7.3 Uncertainty Estimation with Larger Ensembles
Prior works using ensembles of explicit models to estimate epistemic uncertainty [44, 12, 10] typ- ically employ larger ensembles of 2 5 models, whereas IIFI. uses 2. We wish to evaluate the impact of this smaller number of models. However, the Jeffreys divergence is only defined for two distributions, and while other divergence measures (e.g. densen-Shannon, can be generalized to an arbitrary number of distributions, they typically require knowledge of the intractable partition functions of the distributions. Accordingly, we consider estimating the uncertainty of implicit models by computing the average of the Jeffreys divergences between every pairwise combination of models. Figure 5 provides intuition this measure, and we provide information computation time in Section 7.4.4.
evaluate the effect of adding more models by comparing the estimate of the Jeffreys divergence with n 2 models and the averaged estimate with = 5 models the 1,2 distance between the robot policy's proposed action and the expert policy's action at the same state. While ground truth epistemic uncertainty is intractable to calculate, the ground truth action discrepancy between the human and robot can provide correlate of uncertainty: higher discrepancy corresponds to higher uncertainty. The results are shown in Figure 6. We observe that both ensemble sizes are positively correlated with action discrepancy, and that the ensemble with n 5 models has a higher correlation (r = 0.804) than the ensemble with n = models (r = 0. 0.688). We also observe that the n ensemble has lower variance than 2: the standard deviation is 0. 176 compared 0.220. These results suggest that larger ensembles can improve the uncertainty estimation the of increased computation time (2.6x in Section 7.4.4).
7.4 Additional Experimental Details
7.4.1 IFL Benchmark Hyperparameters
Implementations Implicit Interactive Fleet Learning and baselines are available in the code sup- plement and are configured to run with the same hyperparameters we used in the experiments. To compute the uncertainty thresholds for Explicit IFI. and IIFI. (see Section 8.3.1 in [13] for defini- tion), we Explicit BC and Implicit BC respectively with N 100 robots for 1000 timesteps and choose the 99th percentile value among all 100 1000 uncertainty values. The Srank environment sets these thresholds to zero since there are no constraint violations (i.e., this sorts robot
13

---

Energy Functions
.40
24
4.7
1080
-10.59
100.01
13
0.04
-0.04
-220.04
100
61
-102.08
-0.06
-100.02
-0.00
-107.08
-0.00
-0.30
40.10
-100.10
-0.30
-100
244
-0.34
-0.14
-102.14
-0.14
-100.11
Site
(a) Consider 5 isotropic Gaussian energy functions, each a negative Gaussian probability density function with some offset.
Aousaaee of Pairninal jels Divanganaas
0.39
0.334
0331
(b) We numerical integration calculate at each state the Jeffrreys divergences be- tween each the (3) 10 unique pairs of models, and report the average value. intuition suggests, the calculated uncertainty highest at states and where two of the Gaussians have means that are far apart, meaning that they strongly prefer very different actions. At state 0, the uncertainty is lower as model strongly prefers the action 0, and the others closer uniform. Far from state 0, the uncertainty is lowest as the energy functions are approximately flat.
Figure
14

---

1.4
n=2 estimate
n=5 averaged estimate
1.2
Estimated Jeffreys divergence
1.0
0.6
0.2
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
L2 distance between expert and agent actions
Figure 6: We plot the Jeffreys divergence estimates and the ground truth action discrepancies at the first 1000 states visited by robot with unimodal policy. Both variants of the Jeffreys divergence calculation are posi- tively correlated the correlation coefficient is 0.688; in the n = case, the correlation coefficient is r 0.804, indicating that additional models can make the ensemble more predictive of when the agent will deviate from the expert (at the cost of increased computation time).
L,2 distance between the robot policy's 's and expert policy's actions.
n case,
priority by uncertainty alone). See Table for these values, state and action space dimensionality. and other hyperparameters. The batch size 512 and all algorithms pretrain the policy for N/2 gradient steps, where N the number of data points in the 10 offline task demonstrations. Finally, in prior work [13), the Random IIFL. baseline is given human action budget that approximately equals the average amount of human supervision solicited by IIFL. See the code for more details.
Environment
S
|
Explicit ú
Implicit à
BallBalance
24
3
0.1179
0.1206
Ant
60
8
0.0304
0.9062
Anymal
48
12
0.0703
2.2845
CuaetaC
19
7
0.0
0.0
Table 4: Simulation environment hyperparameters.
7.4.2 FrankaCubeStacc Environment
The scripted supervisor for FrankaCubeStacc is defined in human action() of auv/isaacgym/frraka..a www.aaaubbbbeeceeeeeeeeeeee py in the code supplement. Using known pose infor- mation and Cartesian space control, the supervisor policy does the following, where Cube A is to be stacked Cube B: (1) move the end effector
position above Cube A; (2) rotate into pre-grasp pose; (3) descend to Cube A; (4) lift Cube A; (5) translate to a position above Cube B; (6) place Cube A on Cube B; and (7) release the gripper. Heterogeneity is concentrated Step 2: while one supervisor rotates to an angle E [0, that corresponds to a pair of antipodal faces of the cube, the others rotate to f म, ,, and f + 3. See Figure 7 for intuition. We also include results for only hetereogeneous policies (0 and suggest that relative performance of IIFL over baselines remains approximately consistent as the number of modes varies and can improve multimodality increases.
3) in Table results (in conjunction with Table 1)
15

---

Figure 7: The scripted heterogeneous supervisors for the FrankaCubeStack Isaac Gym environment pick dif- ferent faces cube for the same cube pose.
Algorithm
Avg. Reward
Task Successes
ROHE
BC
23.45 = 0.99
0.0 0.0
N/A
IBC
30.32 + 2.78
0.0 + 0.0
N/A
IFL.
307.87 118.59
9.3 4.7
3.08 1.19
IIFL-R
244.98 + 32.58
0.0 0.0
2.45 + 0.33
IIF
604.17 = 263.06
17.7 11.1
6.04 + 2.63
Table 5: Execution results from the FrankaCubeStack Isaac Gym environment with heterogeneous supervisor policies (rather than 4).
7.4.3 Physical Experiment Protocol
We largely follow the physical experiment protocol in Hoque al. [13] but introduce some modifi- cations human supervision. We execute 3 trials of each of algorithms (Explicit BC, Implicit BC, Explicit IFL, Implicit IFL) the fleet of 4 robot arms. Each trial lasts 150 timesteps (synchronous across the fleet) for total of 3 x 4 x 4 x 150 7200 individual pushing actions. The authors provide human teleoperation and hard resets, which differ from prior work due to the continuous action space and the square obstacle in the center the workspace. Teleoperation done using an OpenCV (https://penccv.og)) GUI by clicking on the desired end point of the end-effector in the overhead camera view. Hard resets are physical adjustments of the cube to randomly chosen side of the obstacle. IIFL is trained online with updated data at 50 and 100 while IFL is updated at every timestep (with equivalent total amount of gradient steps) to follow prior work [13].
The rest of the experiment protocol matches Hoque et al. [13]. The 2 ABB YuMi robots are located about 1 km apart; a driver program uses the Secure Shell Protocol (SSH) to connect to machine that connected the robot via Ethernet, sending actions and receiving camera observations. Pushing actions are executed concurrently by all 4 arms using multiprocessing. We minimum intervention time and hard reset time
5. All policies image-action pairs (336 samples collected by the authors with 10x data augmentation). 10x data augmentation on the initial offline dataset as well as the online data collected during execution applies the following transformations:
initialized with an offline dataset of 3360
Linear contrast uniformly sampled between 85% and 115%
Add values uniformly sampled between -10 and 10 to each pixel value per channel
Gamma contrast uniformly sampled between 90% and 110%
Gaussian blur with uniformly sampled between 0.0 and 0.3
Saturation uniformly sampled between 95% and 105%
Additive Gaussian noise with uniformly sampled between 0 and 86
255 80 x 255
16

---

7.4.4 Computation Time
In Table we report the mean and standard deviation of various computation time metrics. All timing experiments were performed with N 100 robots and averaged across T 100 timesteps in the Ant environment on single NVIDIA Tesla V100 GPU with 32 GB RAM. Training time is reported for a single gradient step with a batch size of 512. Note that with default hyperparameters, IFL trains an ensemble of 5 (explicit) models and IIFL trains an ensemble of 2 (implicit) models; hence, we also report the training time per individual model. IFL inference consists of a single forward pass through each of the models, while IIFL inference performs 100 Langevin iterations; both of these vectorized across all 100 robots once. IFL uncertainty estimation also consists of single forward pass through each of the 5 models while IIFI. performs both Langevin iterations and 2 forward passes through each of the 2 models. While IIF can provide policy performance benefits over IFL, we observe that it comes with a tradeoff of computation time, which may be mitigated with parallelization across additional GPUs. Furthermore, while uncertainty estimation is the bottleneck in IIFL, it is performed with sub-second latency for the entire fleet. This is significantly faster than alternatives such as directly estimating the partition function, which both less accurate and slower; we measure take an average of 7.10 seconds per step using annealed importance sampling [48]. Finally, uncertainty estimation for the variant described in Section 7.3 that uses
implicit models required 2.599=1.002ss. While the time complexity should grow as quadratic in n, in practice we observe that for small values of n the growth is closer to linear as the latency is dominated by the (n)) sampling process rather than the (n²²) forward passes.
Time
IFL
IIFL
Training step (s)
0.0385 + 0.0205
0.694 + 0.207
Training step per model (s)
0.0077 = 0.0041
0.347 = 0.104
Inference (s)
0.0060 = 0.0395
0.494 + 0.045
Uncertainty estimation (s)
0.0029 = 0.0008
0.988 = 0.008
Table 6: Computation times training, inference, and uncertainty estimation for IFI. IIFI..
17


Reviews:
[{"review_id": "VmTOg19OiG8", "reviewer": "Reviewer_XAiu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents implicit interactive fleet learning (IIFL), an approach that mitigates the impact of distribution shift and multimodality on robot policies learned via imitation learning. The general idea is to combine implicit policies with interactive fleet learning (a state-of-the-art approach where multiple supervisors are available to provide corrective feedback during learning). While the implicit policies allow for heterogeneous supervision (multimodality), interactive learning helps to mitigate distribution shifts. The paper also proposes to use Jeffrey's uncertainty estimate as the criterion to determine which robots should be first assigned to a human supervisor. Experiments in simulation and with a physical robot show that IIFL results in improved robot performance (success) and return on human effort.\n", "summary_of_recommendation": "This is an interesting paper with a potential impact on learning more robust and diverse robot policies from human demonstrations, however, the current evaluation provides limited evidence and few details on how much multimodality and/or distributional shift can be handled by the proposed approach before performance starts to degrade.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "For instance, in the case of the ball balance, ant and animal benchmark, how are distribution shifts introduced during testing? Also, it is not clear why IIGL results in a higher number of hard resets for the Anymal environment.\n\nExperiments focused on showcasing how IIFL can handle multimodality are limited to 2 possible demonstrators. It would be interesting to show how IIFL's performance changes as the number of heterogeneous supervisors increases (using the FrankaCubeStack task for example).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of combining two existing approaches (implicit policies and interactive learning) as a mechanism to address two of the key challenges in imitation learning is interesting. The paper is well-written, the related work does a good job of motivating the proposed idea and its relevance to robotic applications, and the mathematical exposition is clear and well-presented. The inclusion of a physical robot experiment as part of the evaluation is also greatly appreciated.\n\nThe main weakness of the paper in its current state relates to the “Experimental Results” section. Specifically, although the chosen experiments try to showcase how IIFL can mitigate the effects of distribution shift and multimodality, the current description does not provide clear details on how severe these two issues are nor how IIFL scales to more complex setups. \n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-VB7-1ldo9", "reviewer": "Reviewer_9GiX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method for interactive learning of robotic policies from multi-modal human demonstrations. It uses implicit behavior cloning to represent a learned policy as an energy model, where actions can be extracted by optimizing over it. As the policies are learned on a fleet of multiple robots, the interactive part is based on uncertainty prioritized DAgger-style method where a robot with the highest uncertainty is chosen for humans to provide demonstrations. The paper introduces a novel way to estimate epistemic uncertainty by using the Jeffreys divergence metric to estimate the distance between two distributions from two energy models. The method is tested on simulated tasks from the Isaac Gyms suite and a real world block pushing task on 4 robotic arms from 2 ABB Yumi robots. The method is shown to outperform non-interactive baselines, such as Behavioral Cloning (BC) and Implicit BC, and Interactive Fleet Learning (IFL) baseline that did not use implicit policies.", "summary_of_recommendation": "The paper introduces a novel way to estimate uncertainty with implicit policy models and deploys it for a an interactive data collection on multiple agents. The paper is interesting but some more experiments on a wider range of real world tasks would help better understand the general applicability of the method.\n\nUPDATE after rebuttal:\nThank you for new experiments, I increased my score.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please refer to the review for rebuttal questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The problem of interactive learning from human demonstrations that often can be multi-modal is important for scaling up robot learning and deploying robots for an effective data collection.\n- Using Jeffreys divergence is a particularly interesting way to estimate epistemic uncertainty and can be used for a wide range of tasks.\n- The paper is well-written and easy to understand and follow.\n- The method is shown to outperform baselines in simulation and also work in the real world on multiple robotic arms.\n\nWeaknesses:\n- In the paper, the method uses 2 implicit models per policy to compute uncertainty. It would be useful to see an ablation on whether more models can improve this estimation (at a cost of increased compute requirements) as it is not completely clear how well uncertainty can be estimated from such a small amount of models.\n- Generally it would be interesting to see a wider range of real world experiments on more articulated tasks with a wider variety of objects to better understand the general applicability of the method and how well implicit models can estimate uncertainty in more complex environments.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-A-U9X7U8qR", "reviewer": "Reviewer_z8az", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces Implicit Interactive Fleet Learning (IIFL), a novel imitation learning algorithm that addresses both distribution shift and multimodality issues, common in conventional methods. IIFL integrates implicit models with interactive fleet learning and applies Jeffreys divergence to estimate uncertainty in energy-based models. The method demonstrates impressive performance, delivering a 4.5x higher return on human effort in simulation experiments and an 80% higher success rate in a physical block-pushing task.", "summary_of_recommendation": "In conclusion, while the paper offers valuable insights into addressing multimodality and distribution shift in imitation learning, I propose a weak rejection at this stage due to my concerns about the orthogonality of these problems and the perceived limited novelty of the A+B approach. However, I am open to reconsidering my rating based on the authors' replies.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Given the distinct nature of multimodality and distribution shift in imitation learning, could you elaborate on the decision to combine both elements in the IIFL approach and its impact on the novelty of the solution?\n2. How does the IIFL approach tackle the sparsity of human feedback, given that its current design seems to rely on the availability of dense and consistent feedback to form the distribution?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper is well-written and provides a clear description of the setup and implementation details.\n2. The novel integration of implicit Behavioral Cloning (BC) and human-in-the-loop imitation learning presents a fresh perspective. The usage of symmetric KL divergence to facilitate an implicit policy that adapts to human supervision is straightforward and neat.\n3. The experimental results demonstrate significant performance enhancements, with an impressive 80% improvement in real-world robot tasks.\n\nWeaknesses:\n1. While I appreciate the endeavor, my main concern with this work lies in the handling of multimodality and distribution shift, which, in my opinion, represent two orthogonal challenges for imitation learning. While it's reasonable to fuse these elements for optimal imitation learning performance, it resembles more of an A+B solution with restricted novelty. A more natural focus could be on interactive learning from multiple human supervisors, as different supervisors may offer varied feedback - hence the necessity for an implicit model to manage such multimodal feedback. If the multimodality in this work stems from task design, offering multiple solutions to the goal as indicated in Figures 1 and 4, then it seems to be a subject independent of interactive learning.\n2. The experimental outcomes presented in the first image of Figure 3 appear weird, suggesting that the method with interactive human feedback underperforms the BC baseline. This could imply that interactive learning might not always enhance imitation learning. It would be beneficial for the authors to provide an explanation for this outcome, elaborating on potential underlying factors.\n3. The experiments are mostly ablation studies and lacks comprehensive comparisons with recent works in interactive learning, such as reference [22].\n4. The real-world experiments seem overly simplified. The task design fails to showcase the true potential of interactive imitation learning, given that successful pushing results using imitation learning can be achieved without the necessity for interactive learning as is shown in many prior works.\n5. I like the idea of employing symmetric KL divergence to fine-tune the implicit model based on human feedback. However, how does the current approach deal with the sparsity lies in the human feedbacks? The existing method seems to rely on an assumption that dense and consistent human feedback will be available to shape the distribution. It might be beneficial to explore ways to increase sample-efficiency and manage sparse feedback that encompasses diverse corner cases, especially in an interactive learning setting with multiple supervisors.\n6. While the proposed method has the capacity to manage multiple human supervisors, the experiments limits itself to just two. It raises the question whether the inclusion of more supervisors might present any complications to the algorithm.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduced an implicit interactive fleet learning method, combining implicit policy learning into the interactive fleet learning framework. It aims to address the multi-human multimodality and the distribution shift when the robot fleet learns from multiple human supervisors. A key innovation behind this work is using Jeffrey's uncertainty estimate to determine which robots should be first assigned to a human supervisor. At the initial reviews, some clarification questions and concerns about the novelty were raised. The authors did a great job in their rebuttal, addressing most of the issues brought up by the reviewers. Toward the end of the discussion period, all three reviewers felt positive about this work. The AC agreed with Reviewer z8az that "The real-world experiments seem overly simplified." This work could be made much stronger if more complex tasks had been evaluated with the physical robot. Nonetheless, this work has offered a convincing solution to an important problem in interactive learning. The AC supports the reviewers' recommendations to accept this paper at CoRL.
Decision: Accept (Poster)