ASID: Active Exploration for System Identification in Robotic Manipulation
Authors: Marius Memmel
Andrew Wagenmaker
Chuning Zhu
Dieter Fox
Abhishek Gupta
Venue: ICLR 2024 oral (2024)
Keywords: sim2real
system identification
exploration
Primary area: applications to robotics, autonomy, planning

Abstract: Model-free control strategies such as reinforcement learning have shown the ability to learn control strategies without requiring an accurate model or simulator of the world. While this is appealing due to the lack of modeling requirements, such methods can be sample inefficient, making them impractical in many real-world domains. On the other hand, model-based control techniques leveraging accurate simulators can circumvent these challenges and use a large amount of cheap simulation data to learn controllers that can effectively transfer to the real world. The challenge with such model-based techniques is the requirement for an extremely accurate simulation, requiring both the specification of appropriate simulation assets and physical parameters. This requires considerable human effort to design for every environment being considered. In this work, we propose a learning system that can leverage a small amount of real-world data to autonomously refine a simulation model and then plan an accurate control strategy that can be deployed in the real world. Our approach critically relies on utilizing an initial (possibly inaccurate) simulator to design effective exploration policies that, when deployed in the real world, collect high-quality data. We demonstrate the efficacy of this paradigm in identifying articulation, mass, and other physical parameters in several challenging robotic manipulation tasks, and illustrate that only a small amount of real-world data can allow for effective sim-to-real transfer.

Published as a conference paper at ICLR 2024
ASID: ACTIVE EXPLORATION FOR SYSTEM IDENTIFI- CATION IN ROBOTIC MANIPULATION
Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Dieter Fox, Abhishek Gupta Paul G. Allen School of Computer Science & Engineering
University of Washington Seattle, WA 98195, USA {memmelma, ajwagen, zchuning, fex, abhgupta]@cs sttgugraajes...stiigroo.... edu
ABSTRACT
Model-free control strategies such as reinforcement learning have shown the abil- ity learn control strategies without requiring an accurate model or simulator the world. While this is appealing due to the lack of modeling requirements, such methods can be sample inefficient, making them impractical in many real-world domains. On the other hand, model-based control techniques leveraging accurate simulators can circumvent these challenges and use a large amount of cheap sim- ulation data to learn controllers that effectively transfer to the real world. The challenge with such model-based techniques is the requirement for an extremely accurate simulation, requiring both the specification of appropriate simulation as- sets and physical parameters. This requires considerable human effort to design for every environment being considered. tem that can leverage a small amount of real-world data to autonomously refine a simulation model and then plan an accurate control strategy that can be deployed in the real world. Our approach critically relies on utilizing an initial (possibly inaccurate) simulator to design effective exploration policies that, when deployed in the real world, collect high-quality data. We demonstrate the efficacy of this paradigm in identifying articulation, mass, and other physical parameters in sev- eral challenging robotic manipulation tasks, and illustrate that only small amount of real-world data can allow for effective sim-to-real transfer. Project website at https: / i rd abuw. gi thub. io/asid
this work, we propose learning sys-
INTRODUCTION
Learn Explore Randomized Sim
Explore Collect Data Real
Controlling robots to perform dynamic, goal-directed behavior in the real world is challenging. Reinforcement Learning (RL) has emerged as a promising technique to learn such behaviors without requiring known models of the environment, instead relying on data sampled directly from the environment (Schulman et al., 2017a; Haarnoja et al.) (018). In principle, these techniques can be deployed in new environments with minimal amount of human effort. and allow for continual improvement of behavior. Such techniques have been shown to successfully learn complex behaviors in variety of scenarios, ranging from table-top manipulation (Yu et al.) 2020) to locomotion (Hwangbo et 2001)) and even dexterous manipulation (Zhu et al. 2019).
Randomized Puck Mass & Pisic
System Identification
Solve the Task Real
Solve Task Accurate Sim
3
"Sheot Wike zone"
Figure 1: ASID: A depiction our proposed process of active exploration for system identification, from learning exploration policies to real-world deployment.
at cost, requiring full access to the environment in order to de- sign reset mechanisms and reward functions. Such requirements necessitate training these methods in carefully controlled and often eurated environments, limiting their applicability. Moreover, de-
However, these capabilities

---

Published as a conference paper at ICLR 2024
ploying tabula-rasa reinforcement learning methods in the real world often requires prohibitively large number of samples, which may be impractical to collect.
One approach to circumvent these challenges is to rely on simulators to cheaply generate large amounts of data and use RL to train a policy. However, directly deploying policies trained in sim- ulation in the real world is often ineffective due to the discrepancy between the simulation and the real world, the so-called sim2real gap. For example, the physical properties and parameters of the simulation might diverge from the real world, rendering simulation-trained policy useless in reality.
Taking inspiration from system identification, we argue that the key to effective sim2real transfer is an initial round of exploration in the real world learn an effective simulator. We propose generic pipeline for sim2real transfer, Active Exploration for System IDentification (ASID). which decou- ples exploration and exploitation: (I) exploration in real to collect informative data of unknown parameters, (2) refinement of the simulation parameters using data collected in real, (3) policy train- ing on the updated simulator to accomplish the goal tasks. Our exploration procedure is motivated by work in theoretical statistics and seeks to induce trajectories corresponding to large Fisher in- formation, thereby providing maximally informative observations. By using our initial round of exploration to obtain accurate estimates of the parameters in the real world, we show that in many cases, the policies trained in step (3) successfully transfer to real in a zero-shot fashion, even in settings where training policy in sim without additional knowledge of real would fail.
A key insight in our approach is that, while a policy trained in sim to accomplish the goal task may not effectively transfer, strategies that explore effectively in sim often also explore effectively in real. As an example, say our goal task is to hit a ball to a particular location with a robotic arm and assume the mass of the ball is unknown. If train policy in sim hit the ball without knowledge of the mass, when deployed in real it will almost certainly fail, as the force at which it should strike the ball depends critically on the (unknown) mass. To learn the mass, however, essentially any contact between the ball and the arm suffices. Achieving some contact between the ball and the arm requires significantly less precise motion, and indeed, does not require any prior knowledge of the mass. We can therefore train a policy in sim that learns to effectively explore- hit the ball in any direction- and deploy this in real to collect information on the true parameters, ultimately allowing us to obtain a higher- fidelity simulator that does allow sim2real transfer on the goal task.
We are particularly interested the application of our pipeline to modern robotic settings and eval- uate ASID on four tasks: sphere manipulation, laptop articulation, rod balancing. and shuffleboard. We show that in all settings, our approach is able to effectively identify unknown parameters of the real environment (c.g. geometry, articulation, center of mass, and physical parameters like mass, friction, or stiffness), and using this knowledge, learn a policy in sim for the goal task that success- fully transfers real. In all cases, by deploying effective exploration policies trained in simulation, we require only a very small amount of data from real tayy-typccally a single episode of data suffices.
2 RELATED WORK
System Identification:
Our work is closely related to the field of system identification (Aström &
Eykhoff 1971 Söderström & Stoical 1989 Ljung 1998 Schön et al. 2011 Menda et al.) 2020), which studies how to learn a model of the system dynamics efficiently. A large body of work,
stretching back decades, has explored how inputs to system should be chosen most effectively
learn the system's parameters (Mehra 1974 1976 Goodwin & Payne
1977; Hjalmarsson et al.)
1996 Lindqvist& & Hjalmarsson 2001] Gerencsér & Hjalmarsson|] 2005
Rojas al.) |007] Gevers
et al. 2009 Gerencsér et al.] 2009) Manchester]] 2010 Rojas et al.] 201 Bomboois et al., 2011
Hägg al. 2013 Wagenmaker
Jamieson, 2020; Wagenmaker et al. 2021 Mania et al. 2022
Wagenmaker et al.) 2023) or how to deal with partial observability (Schön et al. 2011,MMende et al.
2020). Similar to our exploration strategy, many of these works choose their inputs to maximize some function of the Fisher information matrix. A primary novelty of our approach is to use simu- lator to learn effective exploration policies, and to apply our method to modern, real-world robotics tasks-indeed our work can be seen as bridging the gap between classical work on system identifi- cation and modern sim2real techniques. While the aforementioned works are primarily theoretical, recent work has studied the application of such methods to variety of real-world settings like active
identification of physics parameters (Xu et al.] 20119] Kumar et al.] 2019: Mavrakis et al. 2020 Gao
et al. 2020 2022) kinematic structure (Mo et al. 2021
Wang al. 2022
Nie et al.
2022 Hsu
2

---

Published as a conference paper at ICLR 2024
et al. 2023) through object-centric primitives. Another line of recent work aims to learn the parame- ters of the simulator to ultimately train a downstream policy on the learned parameters, and therefore apply task-specific policies for data collection (Zhu al., 2018 Chebotar et al.. 2019 Huang al. 2023 Ren et al., 2023) or exploration policies that minimize its regret (Liang et al. (020). The ma- jority of these works, however, do not consider running exploration policy that targets learning the unknown parameters, do not address solving downstream tasks, or rely on techniques that do not scale effectively to more complex tasks.
Transferring learned policies from sim2real has shown to be suc- cessful in challenging tasks like dexterous manipulation (OpenAl et al. 2018 Handa et al. 2022 Chen et al. [022)), locomotion (Rudin et al., (002)) agile drone flight (Sadeghi & Levine 2016)
cotuulattonttoReallll Transfer:
or contact rich assembly tasks [Tang et al. (2023). yet challenges remain due to the sim2real gap. To deal with the gap, Domain Randomization (DR) (Tobin et al. 2017) trains policies over a dis- tribution of environments in simulation, hoping for the real world to be represented among them.
Subsequent works adaptively change the environment distribution (Muratore et al. 2019; Mehta
ct al. 2020) and incorporate real data (Chebotar et al. 2019 Ramos et al. [019]] Duan et al. 2023
Chen et al. 2023 Ma et al. 2023 Torne et al. 2024). While similar to our approach, these methods
do not perform targeted exploration in real to update the simulator parameters. Other approaches
seek infer and adapt simulation parameters during deployment (Kumar et al.
2021 Qi al.)
2023 Margolis et al. 2023), leverage offline data (Richards et al.) [202]] Booe et al.
2024). or adapt
online (Sinha et al., 2002);; ; in contrast, we do not learn such an online adaptation strategy, but rather a better simulator. Finally, a commonly applied strategy is to train a policy in sim and then fine-tune in the real environment (Julian et al. 2021; (Smith et al., 2022 Nakamoto et al.) 2023); in contrast, we are interested in the (more challenging) regime where a direct transfer is not likely to give any learning signal to fine-tune tune from.
Model-Based RL: Model-based RL (MBRL) aims to solve the RL problem by learning model of the dynamics, and using this model to either plan or solve a policy optimization problem (Deisen- roth & Rasmussen] 2011 Williams et al. 2017 Nagabandi et al. [018 Chua et al. 2018 Janner ctt al. 2019 Hafner et al. 2019 2020; Janner et al. 2022 Zhu et al. [023)) While our approach model-based in some sense, the majority of work in MBRL focuses on fully learned dynamic models; in contrast, our "model" is our simulator, and we aim to learn only a very small number of parameters, which can be much more sample-efficient Furthermore, MBRL methods typically do not perform explicit exploration, while a key piece of our approach is a targeted exploration proce- dure. The MBRL works we are aware of which do rely targeted exploration (Shyam al. 2019) Pathak et al. 2019) typically rely on fully learned dynamic models and apply somewha hat different exploration strategies, which we show Appendix A.3.2 can perform significantly worse.
3 PRELIMINARIES
We formulate our dacision-making setting as Markoy Decision Processes (MDPs). An MDP is defined as tuple M* = (S,A, [Pr)# J Po. (ra)# ) where S is the set of states, A the set of actions, Pa : S x A + As the transition kernel, Po E As the initial state distribution, and Th S A + IR the reward function. We consider the episodic setting. At the beginning of an episode, the environment samples a state $1 ~ Po. The agent observes this state, plays some action a A. and transitions to state 82 (. 81, a1), receiving reward r (1), After H steps, the environment resets and the process repeats. Our primary goal is to learn a policy π-a mapping from states actions- that maximizes reward in the true environment. We denote the value of a policy by V := Esr., . |E Th an),, where the expectation is over trajectories induced playing policy on MDP M We think of the reward r as encoding our downstream task, and our end goal is to find a policy that solves our task, maximizing V We denote such policies as Ttask-
In the sim2real setting considered in this work, we assume that the reward is known, but that the dynamics of the real environment, p* = {Pr) i Jh are initially unknown. However, we assume that they belong to some known parametric family P : (Po 0 E θ). so that there exists some 0 E ④ such that p = Pg.. Here we take A to be some unknown parameter (for example, mass, friction, etc.), and Po the dynamics under parameter 0 (which we might know from physics, first principles, etc.). For any 0 and policy n, the dynamics Po induce a distribution over state- action trajectories, (81, 01, 82...., 8H, aH). which we denote by po(: m). We can think of
3

---

Published as a conference paper at ICLR 2024
Keap
Explore Real
System Identification
Waak Sim
Transfer to Real
Identity center mass and pose Update simulation
Randomize
Interaction data
"Balance the rod"
zero-shot!
Figure 2: Overview of ASID: (1) Train an exploration policy Texp that maximizes the Fisher information, leveraging the vast amount of cheap simulation data. (2) Roll out Texp in real to collect informative data that be used to (3) system identification identify physics parameters and reconstruct, e.g., geometric, collision, and kinematic properties. (4) Train sseciic policy Ttack in updated simulator (5) zero- shot transfer Taask to the real world.
our simulator as instantiating pe(- -)- assume our simulator is able to accurately mimic the dynamics of MDP with parameter 0 under policy х, generating samples po .))
In addition, we also assume that samples from our simulator are effectively "free" for any 0 and generate many trajectories
policy
po(- TT) as wish. Given this, it is possible to find the optimal policy under 9 by simply running any standard RI. algorithm in simulation. With knowledge of the true parameters 0 trajectories from the simulated environment with parameter 0" It follows that, if we can identify the true parameter 0 in real, can solve the goal task.
then easily find the optimal policy in real by sampling
We consider the following learning protocol:
Learner chooses exploration policy Texp and plays it in real for a single episode, generating trajectory Treal ~ pe- (- Txpp).
Using Treal and the simulator in any way they wish, the learner obtains some policy Ttask-
Learner deploys Frask in real and suffers loss max, Vi" -V Vraa
The goal of the learner is then to learn as much useful information as possible about the real en- vironment from single episode of interaction and use this information obtain policy that can solve the task in real as effectively as possible.
Parameter Estimation and Fisher Information: The Fisher information matrix plays key role in the choice of exploration policy, Texpp. Recall that, for distribution po. satisfying certain regularity conditions, the Fisher information matrix defined as:
I(0)= Ex loggPo( Ve loggpo(T)
.....,,7 T. and let ((2) denote some unbiased estimator of 0*. Then the Cramer-Rat lower bound (see e.g. Pronzato & Pázman (203)) states that, under certain regularity conditions, the covariance of O(D) satisfies:
Assume that we have access to data D = (T, Y where T, ~ Pa- for L = 1,...,
ED-pon ( (0) (θ()) - 0*) = 2 I((-)-
From this it follows that the Fisher information serves as a lower bound on the mean-squared error: [(0 0*
Ep [|| [10())
0* || tr tr(Ep-pp
0*) 2 T-1 tr( (1)
This is in general tight- for example, the maximum likelihood estimator satisfies ! with equality as T + oo (Van der Vaart (2000). The Fisher information thus serves as a fundamental lower bound on parameter estimation error, key motivation for our exploration procedure.
ASID: TARGETED EXPLORATION FOR TEST-TIME SIMULATION CONSTRUCTION, DDENTIFICTION,, AND POLICY OPTIMIZATION
In this section, we present our proposed approach, ASID, a three-stage pipeline illustrated in Fig- We describe each component of ASTD in the following.
2
4

---

Published as a conference paper at ICLR 2024
4.1 EXPLORATION VIA FISHER INFORMATION MAXIMIZATION
As motivated in Section 3 to learn a policy effectively accomplishing our task, it suffices to accu- rately identify 0*. In the exploration phase, step 1 in our learning protocol, our goal is to then play an exploration policy Texp which generates a trajectory on the real environment that provides as much information on 0 as possible. Following Section gl the Fisher information gives a quantification of the usefulness of the data collected, which motivates our approach.
In our setting, the distribution over trajectories generated during exploration in real, Treal Po- Texp), depends on the exploration policy, Texp: being played. As the Fisher information depends on the data distribution, too scales with the choice of exploration policy:
I(O Fexp) == E Trrp (-1Nexp)
[o log Mogge- (T Texpp) Vo log po- (T Texp))
Following I if we collect trajectories by playing Texp and set f to any unbiased estimator of 0 on these trajectories, the mean-squared error of 0 will be lower bounded by Ir(I(0", Texp) The optimal exploration policy-the exploration policy which allows for the smallest estimation error- is, therefore, the policy which solves
arg min, tr tr(I(0',
As intuitive justification for this choice of exploration policy, note that the Fisher information is defined in terms of the gradient of the log-likelihood with respect to the unknown parameter. Thus, if playing some Texp makes the Fisher information "large", making tr(I(0", Texp) small, this suggests Texp induces trajectories that are very sensitive to the unknown parameters, i.e., trajectory that are significantly more likely under one set of parameters than another. By exploring maxi- mize the Fisher information, we, therefore, will collect trajectories that are maximally informative about the unknown parameters, since we will observe trajectories much more likely under one set of parameters than another. Motivated by this, we therefore seek to play policy during exploration that solves (
Implementing Fisher Information Maximization: In practice, several issues arise in solving 2). which we address here. First, the form of I(O, .T can be quite complicated. depending on the structure of po(. ir), and it may not be possible efficiently obtain solution ( To address this, we make a simplifying assumption on the dynamics, that our next state, 8h+1, evolves as:
Sh+1 fo Sh. an) + Wh:
where sy and ag are the current state and action, wh ~ (O, ou I) is Gaussian process noise, and fe are the nominal dynamics. Under these dynamics, the Fisher information matrix reduces to
I(0, m) o (-p
Σ D VoJo(Sh, at) Volo(sh, T
We argue that solving (2) with this form of I(0, TT) is a very intuitive objective, even in cases when the dynamics may not Tollow (3) exactly. Indeed, this suggests that during exploration, we should aim to reach states for which the dynamics fe have a large gradient with respect to statts for which the next state predicted by the dynamics very sensitive to O. In such states, observing the next state gives significant amount of information on O, allowing us to accurately identify 0*.
second challenge solving (2) is that we do not know the true parameter 0*, which the opti- mization (2) depends on. To circumvent this, we rely domain randomization in choosing exploration policy, solving instead:
Texp arg min, 10 (tr(I((, T)
(4)
for some distribution over parameters qo- While this is only an approximation of (2) in practice we find that this approximation yields effective exploration policies since, as described in Section in many cases exploration policies require only coarse model of the dynamics, and can therefore often be learned without precise knowledge of the unknown parameters.
A final challenge is that, in general, we may not have access to a differentiable simulator, and our dynamics themselves may not be differentiable. In such cases, Vofo ()) is unknown or
In the experiment design literature, this is known A-optimal experiment design Pukelsheim 2006
5

---

Published as a conference paper at ICLR 2024
undefined, and the above approach cannot be applied. As a simple solution to this, we werely finite- differences approximation to the gradient, which still provides an effective measure of how sensitive the next state is to the unknown parameter. In practice, to solve (4) and obtain exploration policy, we rely on standard policy optimization algorithms, such as PPO (Schulman et al. 2017b).
4.2 SYSTEM IDENTIFICATION
ASID runs the exploration policy Tоx (Section 4.1 in the real environment to generate single trajectory Treal ~ Pg- (- Te In the system identification phase, ASID then updates the simulator parameters using the collected trajectory. The goal is to find distribution over simulator parameters that yield trajectories that match Teal as closely as possible. particular, we wish to find some distribution over simulation parameters, 9s. which minimizes:
Ea-ag [B ppol [A( (A(T)a))) Traal
Tsim
where pe(- A(Trea))) denotes the distribution over trajectories generated by the simulator with parameter 0, and playing the same sequence of actions as were played in Treal. In practice, we apply REPS (Peters et al. 2010) for the simulation and Cross Entropy Method (CEM) for the real- world experiments. We stress that the ASID framework is generic, and other blxck-box optimization algorithms could used instead.
4.3 SOLVING THE DOWNSTREAM TASK
After exploration and identification, the simulator can include information about the kinematic tree, position, orientation, size of the object of interest, and the physical parameters of the real en- vironment. With such a high-fidelity simulator, we aim to solve the downstream tasks entirely in simulation and transfer the learned policies Trask the real world in zero-shot fashion. As with the system identification stage, ASID does not assume particular method for solving the downstream task and any policy optimization algorithm can be used.
5 EXPERIMENTAL EVALUATION
In our experimental evaluation, we aim to answer the following questions:
Does ASID'S exploration strategy yield sufficient information identify unknown parameters?
Are downstream task policies learned via ASID successful when using the identified parameters?
Does the paradigm suggested by ASID transfer to performing tasks in the real world?
We conduct our experimental evaluation in two scenarios. First, we conduct empirical experiments entirely in simulation (Todorov et al.] 2012) to validate the behavior of the exploration, system identification, and downstream task modules of ASID. This involves treating true simulation en- vironment as the real-world environment and then a reconstruction of this simulation environment as the approximate constructed simulation. Policies learned in this approximate simulation can then be evaluated back in the original simulation environment to judge efficacy. Second, we apply this two real-worke manipulation tasks that are inherently dependent accurate parameter identifica- tion, showing the ability of ASID to function in the real world, using real-world data.
5.1 ABLATIONS AND BASELINE COMPARISONS
We compare ASID with several baselines and ablations for various portions of the pipeline.
Exploration: To understand the importance targeted exploration via Fisher information max- imization we compare with two baselines. First, we compare with the naïve approach of using data from random policy for system identification that is not performing any targeted exploration. Secondly, compare with Kumar al. (201)), which aims to generate exploration trajectories that maximize mutual information with the parameters of interest. This method essentially learns a parameter inference network and then rewards exploration policies for minimizing its error.

---

Published as a conference paper at ICLR 2024
(a) Rod balancing
(b) Sphere manipulation
(c) Multi-friction
(d) Articulation
Figure 3: Depiction of environments simulation
System Identification: To evaluate the impact of system identification methods in this pipeline, compare the effectiveness of using optimization-based system identification, e-g.. (Peters al., 2010 Memmel et al. (022), with an end-to-end learned module (Kumar et al., 2019). This compar- ison shows the stability and extrapolation benefits of ASID over completely data-driven modeling techniques. In particular, we evaluate ASID with the optimization- system identification re- placed by the learned estimation module from Kumar al. (2019) (ASID estimator).
Downstream Policy Learning: Since the eventual goal is to solve the downstream task, we finally compare how effective is to learn downstream policies in simulation as opposed an uninformed Domain Randomization (DR) over the parameters, in contrast to the targeted and identified param- eters that stem from the ASID pipeline.
5.2 SIMULATED TASK DESCRIPTIONS
Sphere Manipulation:
We consider two sphere manipulation tasks where physical parameters like rolling friction, object stiffness, and tangential surface friction are unknown and make significant impact on policy performance: 1) striking golf ball to reach goal with unknown system friction (Figure (b).2. 2) striking golf ball in an environment with multiple patches that experience different surface friction (Figure Bc). In each scenario, the position of objects and the physical parameters are varied across evaluations. In this setting, we train downstream tasks with PPO.
Balancing, or dynamic stacking of objects critically depends on an accurate esti- mate of the inertia parameters. We construct rod balancing task where the agent can interact with a rod object (Figure Ba) to identify its physical parameters, in this case varying the distribution of mass along the rod. Once the parameters are identified, the rod must be balanced by placing it ledge (Figure 55 (left). The error is measured by the tilting angle of the rod after placement, a task that requires accurate estimation of system parameters. In this case, downstream policies.
Rod Balancing:
the CEM to optimize
Articulation: To stress generality, consider environments that don't simply identify physical parameters like mass or friction but also the structure of the kinematic tree such as articulation and joint positioning. We consider an environment involving an articulated laptop model with binary environment parameter representing whether articulation is present or not (Figure 3d
5.3 DOES ASID LEARN EFFECTIVE EXPLORATION BEHAVIOR?
To evaluate the exploration behavior quantitatively, we consider the case multiple unknown pa- rameters, where learning each parameter requires exploration in different region. In particular, we compare the exploration policy learned by our method with the exploration method of Kumar et al. (2019), the multi-friction sphere manipulation environment illustrated in Figure 3e where the surface exhibits three different friction parameters (indicated by the grass, sand, and gravel tex- tures). We initialize the golf ball in the leftmost region (grass) to explore, the arm must strike the ball to other regions to identify their friction parameters. We plot heat map of the regions visited by the ball during exploration in Figure 4 As can be seen, our approach achieves roughly uniform coverage over the entire space, learning to strike the ball into each region, and illustrating that our method is able to effectively handle complex exploration tasks that involve exciting multiple pa- rameters. In contrast, the approach of Kumar et al. (2019) does not effectively move the sphere to regions different from starting region.

---

Published as a conference paper at ICLR 2024
Table 1: Downstream task results in simulation: Random exploration fails in tasks where directed explo- ration is required, e-g., striking a sphere or finding the inertia of rod. When placing the rod with single action, domain randomization (DR) cannot solve the task without knowing the physical parameters. Learned system identification (Kumar et all (2019) and ASID + estimator) doesn't t generalize to unseen trajectories and becomes far less effective than optimization based system identification (ef. ASID + SysID).
Task
Rod Balancing
Sphere Striking
Metric
Tilt angle in degree Inertia (middle)
Success Rate in %
Parameter
Inertia (left)
Inertia (right)
Friction ~ [1.1, 1.5)
Random exploration
12.44 + 19.6
4.20 = 6.5
15.34 + 15.9
10.62 + 4.3
Kumar et al. (0199)
13.70
2.82 = 2.7
15.26 9.8
9.50 + 2.4
DR
26.69 7.0
13.05 7.3
1.13 + 1.3
8.75 + 1.5
ASID + estimator
17.73 13.1
4.65 5.1
9.99 6.8
11.00 + 5.2
ASID SysID (ours)
0.00 + 0.0
0.72 1.0
0.00 + 0.0
28.00 + 9.7
We further analyze the exploration behavior in the articulation en- vironment (Figure 3d). Here our exploration policy interacts with
6:32
6:38
0:35
0.13
6:28
0:48
the laptop 80% of the time opposed to 20% for naïve baselines. Appendix A.2.2 shows that models trained to predict articulation from interaction data, e.g., Ditto (Jiang et al. (2022), can infer joint and part-level geometry from the collected data.
5:48
0.00
5.41
6.16
5.35
5:44
0.11
0.43
6:47
220
6-22
6:11
5.18
8:66
0.51
0.59
8:44
0.32
0.04
8:58
0:45
532
5:7
5:88
537
5:37
6:38
0.39
0.36
0.42
(a) ASID
5.4 How DOES ASID PERFORM QUANTITATIVELY IN SIMULATION ON DOWNSTREAM TASKS?
6.27 6.17 0.08
5:34 0.0
0.0
5:06
0.07 0.01
8:0 0:2
0.66
5:31
ca8 100
5.39
0.00
0.05
8.08
0.07
To quantitatively assess the benefits of ASID over baselines and ab- lations, we evaluate the performance of ASID with respect to the baselines in simulation on downstream evaluation for the rod bal- ancing (Figure Bab and sphere manipulation (Figure 3cl task. Re- sults are given in Table [
6.0 1000
0.38
0.11
0.09
8:04
0.06 0.00
a:0
6:37
0.12
8:38
0.11
039
6:38
6:13 0.19
637 0.7
0:18
6.38
0.36
0.18
5.18 6.30 6.15
(b) Kumar et al. 2019
Observing the performance on the rod environment, it becomes clear that previous methods struggle with one-shot tasks. When the inertia is not properly identified, CEM predicts the wrong action with high certainty, causing it to pick and place the rod at the wrong location. Domain randomization (DR) on the other hand tries to learn a policy over all possible mass dis- tributions which in this case leads to policy that always grasps the object at the center of the reference frame. The success here
Figure 4: Visitation frequency of the sphere when explored by different exploration policies on fulti-friction (Figure ASID activates the sphere over much larger area, therehy identifying parameters more accurately depends on "getting lucky" and sampling the correct action for the true inertia parameter.
In the sphere environment, the distance rolled and bounce of the sphere are critically dependent on parameters such as friction and stiffness, and misidentifying these parameters can lead to signifi- cantly different trajectories and unsuccessful downstream behavior. This is reflected in the signifi- cantly higher success rate of ASID compared to baselines that uninformed of parameters and simply optimize for robust policies (DR), those that try to use learned estimators for exploration (Kumar al. 2019) using random exploration.
5.5 DOES ASID ALLOW FOR REAL-WORLD CONTROLLER SYNTHESIS USING MINIMAL REAL-WORLD DATA?
We further evaluate ASID on real-world tasks, replicating the rod balancing task from simulation (Figure 3) and introducing a novel shuffleboard task (Figure As in simulation, we use a Franka Emika Panda robot for exploration and task performance in the real world. We compute the object's position and orientation by color-threshold the pointclouds from two calibrated Intel RealSense D455 tameras-this approach could easily be replaced by a more sophisticated pose estimation system if desired.
The goal of this task is properly balance rod with an unknown mass distri- bution by informatively exploring the environment in the real world and using the resulting data to identify the appropriate physics parameters in simulation. The optimized controller in simula- tion is then deployed to perform the task in the real world. In this case, the downstream task is picking the rod at a certain point along its axis and balancing it on a perch (Figure I Figure 5
Rod Balancing:
8

---

Published as a conference paper at ICLR 2024
The policy executes the downstream task by a pick and place primitive parameterized by the exact pick point. We deploy ASID fully autonomously, executing exploration, system identification, downstream task training, and execution in an end-to-end fashion.
The mass distribution in the rod is varied and both the inertia and the environment friction must be inferred. While ASID correctly iden- tifies the inertia of the rod most of the time, we find that center of mass close to the middle of the rod causes ambiguities that hurt our system identification process causing the simulation to not be accurate enough for zero-shot transfer. Overall, ASID solves the rod-balancing task 6/9 times across varying mass distribution and pose policy trained
Figure 5: Real-wortd Rod Balancing: Simulation setup for training exploration and downstream task poli- cies (left), Successful execution autonomous real- world rod balancing with skewed mass (right).
Table 2:
Downstream task results in real: ASID suc- cessfully balances the rod while domain randomization (DR) fails catastrophically.
Task
Rod Balancing
while a Nomain-randominn without any environment interaction fails to solve it at all (Table
Inertia
left
middle
right
DR
0/3
0/3
0/3
ASID (ours)
2/3
1/3
3/3
Shuffleboard: This task is scaled-down sion of the popular bar game tabletop shuffle-
board where a puck must be shot target area on a slippery board. We closely follow the original game and pour wax (sand) on the board to decrease the surface friction. This modification makes the task especially difficult as the surface friction on the board changes slightly after each shot since the puck displaces the wax. The goal is to strike the puck to one of the target regions-ont one closer (yellow) and one further (blue) away from the robot (Figure 2 ASID estimates the sliding and torsional friction of the puck to account for the changing surface friction of the board. For executing the downstream task, primitive positions the endeffector at a fixed distance to the puck and policy predicts force value that parameterizes a shot attempt.
Figure 6 After exploring the scene,
Due to the changing surface friction, the do- main randomization baseline struggles to shoot the puck to the desired zone. While it succeeds 3/10 times-probbbly because the surface fric- tion was part of its training distribution other attempts fall short overshoot the tar- get. With its dedicated exploration phase, ASID can accurately adapt the simulation to the cur- rent conditions and lands the puck in the desired zone 7/10 times (Table
Figure 6: Real-world Shuffleboard: Simulation setup for training exploration and downstream task poli- cies (left). Successful strike the yellow zone (right).
DISCUSSION
In this work, we introduced novel pipeline for performing real-world control by au- tonomously exploring the environment, using the collected data for system identification, and the resulting identified system for downstream policy optimization. In essence, this sim-to- real-to-sim-to- re-rlll pipeline allows for targeted test-time construction of simulation environ- ments in a way that enables the performance of downstream tasks. We demonstrate how this type of identification requires autonomous and carefully directed exploration, and introduce a novel algorithm based on Fisher information maximization that is able to accomplish this directed explo- ration. The autonomously collected trajectories can then be paired with downstream optimization- based system identification and reconstruction algorithms for accurate simulation construction and downstream policy learning. We show the efficacy of this paradigm on multiple environments in simulation, as well as rod balancing and shuffleboard, two challenging real-world tasks.
Table 3: Downstream task results in real: ASID out- performs domain randomization (DR) shooting the puck to the desired zones.
Task
Shuffleboard
Target zone
yellow (close)
blue (far)
DR ASID (ours)
2/5
1/5
4/5
3/5

---

Published as a conference paper at ICLR 2024
REFERENCES
Karl Johan Aström and Peter Eykhoff. System identification-a survey. Automatica, 1971.
Xavier Bombois, Michel Gevers, Roland Hildebrand, and Gabriel Solari. Optimal experiment design for open and closed-loop system identification. Communications Information and Systems, 11 (3):197-224, 2011.
Avinandan Bose, Simon Shaolei Du, and Maryam Fazel. Offline multi-task transfer rl with repre- sentational penalization. arXiv preprint arXiv:2402.12557 12570, 2024.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In ICRA, 2019.
Qiuyu Chen, Marius Memmel, Alex Fang. Aaron Walsman, Dieter Fox, and Abhishek Gupta. Urd- former: Constructing interactive realistic scenes from real images via simulation and generative modeling. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition CoRI20233, 2023.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:2211.11744, 2022.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leamn ing in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Mare Deisenroth and Carl Rasmussen. Pilco: model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pp- 465-472, 2011.
Jiafei Duan, Yi Ru Wang, Mohit Shridhar, Dieter Fox, and Ranjay Krishna. Ar2-d2: Training robot without robot. In 7th Annual Conference Robot Learning, 2023.
Ziyan Gao, Armagan Elibol, and Nak Young Chong. A 2-stage framework for learning to push unknown objects. In Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 2020.
Ziyan Gao, Armagan Elibol, and Nak Young Chong. Estimating the center of mass of an unknown object for nonprehensile manipulation. In IEEE International Conference on Mechatronics and Automation (ICMA), 2022.
László Gerencsér and Håkan Hjalmarsson. Adaptive input design in system identification. In Pro- ceedings of the 44th IEEE Conference on Decision and Control, pp. 4988-4993. IEEE, 2005.
László Gerencsér, Håkan Hjalmarsson, and Jonas Mårtensson. Identification of arx systems with non-stationary inputs asymptotic analysis with application to adaptive input design. Automatica, 45(3):623-633, 2009.
Michel Gevers, Alexandre S Bazanella, Xavier Bombois, and Ljubisa Miskovic. Identification and the information matrix: how get just sufficiently rich? IEEE Transactions Automatic Con- trol, (AAAAARICE))2888-222228 22440 2009.
Graham Clifford Goodwin and Robert L Payne. Dynamic system identification: experiment design and data analysis. Academic press, 1977.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1011v:1801.01220, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: /0t.::/9/200000 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint 1oXiv:2010.02/99 2020.
10

---

Published as a conference paper at ICLR 2024
Per Hägg. Christian A Larsson, and Häkan Hjalmarsson. Robust and adaptive excitation signal gen- eration for input and output constrained systems. In 2013 European Control Conference (ECC), pp. 1416-1421. IEEE, 2013.
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation reality. arXiv preprint arXiv:2210.13702, 2022.
Häkan Hjalmarsson, Michel Gevers, and Franky De Bruyne. For model-based control design, closed-loop identification gives better performance. Automatica, 32(12): 16591)6336, 1996.
Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In ICRA, 2023.
Peide Huang, Xilun Zhang, Ziang Cao, Shiqi Liu, Mengdi Xu, Wenhao Ding. Jonathan Francis, Bingqing Chen, and Ding Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Conference on Robot Learning, pp. 734-760. PMLR, 2023.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint S0ti::2005009991,, 2022.
Zhenyu Jiang, Cheng-Chin Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In CVPR, 2022.
Ryan Julian, Benjamin Swanson, Gaurav Sukhatme, Sergey Levine, Chelsea Finn, and Karol Haus- man. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. In CoRL, 2021.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0400 107.04044 2021.
K Niranjan Kumar, Irfan Essa, Sehoon Ha. and C Karen Liu. Estimating mass distribution of artic- ulated objects using non-prehensile manipulation. arXiv preprint arXiv:1907.03960 2019.
Jacky Liang, Saumya Saxena, and Oliver Kroemer. Learning active task-oriented exploration poli- cies for bridging the sim-to-real gap. Robotics science and systems, 2020.
Kristian Lindqvist and Hákan Hjalmarsson. Identification for control: Adaptive input design using convex optimization. In Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No. 01CH37228), volume 5, pp. 13226-4331 IEEE, 2001.
Lennart Ljung. System identification. Springer, 1998.
Liqian Ma, Jiaojiao Meng, Shuntao Liu, Weihang Chen, Jing Xu, and Rui Chen. Sim2real?: Actively building explicit physics model for precise articulated object manipulation. In ICRA, 2023.
Ian R Manchester. Input design for system identification via convex relaxation. In 49th IEEE
Conference on Decision and Control (CDC), pp. 2041-2046. IEEE, 2010.
Horia Mania, Michael Jordan, and Benjamin Recht. Active learning for nonlinear system identifi- cation with guarantees. J. Mach. Learn. Res., 23:32-1, 2022.
Gabriel B Margolis, Xiang Fu, Yandong Ji, and Pulkit Agrawal. Learning physically grounded robot vision with active sensing motor policies. CoRL, 2023.
Nikos Mavrakis, Rustam Stolkin, et al. Estimating an object's inertial parameters by robotic pushing: data-driven approach. In IROS, 2020.
11

---

Published as a conference paper at ICLR 2024
Raman Mehra. Optimal input signals for parameter estimation in dynamic systems systems- survey and new results. IEEE Transactions on Automatic Control, 19(6):55--768, 1974.
Raman K Mehra. Synthesis of optimal inputs for mutrinput-multioupuu (mimo) systems with pro- cess noise part Frequene y-domain synthesis part ii: Time-domain synthesis. In Mathematics in Science and Engineering, volume 126, pp. 211-249. Elsevier, 1976.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In CoRL, 2020.
Marius Memmel, Puze Liu, Davide Tatco, and Jan Peters. Dimensionality reduction and prioritized exploration for policy search. In International Conference on Artificial Intelligence and Statistics, 2022.
Kunal Menda, Jean De Becdelievre, Jayesh Gupta, Ilan Kroo, Mykel Kochenderfer, and Zachary Manchester. Scalable identification partially observed systems with certainty-equivaeet em. In ICML. PMLR, 2020.
Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In CVPR, 2021.
Fabio Muratore. Michael Gienger, and Jan Peters. Assessing transferability from simulation real- ity for reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2019.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine. Neural network dynamics for model-bases deep reinforcement learning with model-free fine-tuning. In 2018 IEEE interna- tional conference on robotics and automation (ICRA), pp. 7559-7566. IEEE, 2018
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL:: Calibrated offline RL pre-training for efficient online fine-tuning. In Workshop on Reincarnating Reinforcement Learning at ICI.R, 2023.
Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song, Structure from action: Learning interactions for articulated object 3d structure discovery. arXiv preprint (iXXix:2007009997 2022.
Marcin Andrychowicz OpenAI, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, et al. Learn- ing dexterous in-hand manipulation. arXiv preprint a 1808.00177 2018.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 10062-50711 PMLR, 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI, 2010.
Luc Pronzato and Andrej Pázman. Design of experiments in nonlinear models. Lecture notes in statistics, 212(1), 2013.
Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In CoRL, 2023.
Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv: 017728 2019.
Allen Z Ren, Hongkai Dai, Benjamin Burchfiel, and Anirudha Majumdar. Adaptsim: Task-driven simulation adaptation for sim-to-real transfer. arXiv preprint arXiv:2302.04999. 2023.
SM Richards, N Azizan, J-JE Slotine, and M Pavone. Adaptive-conttoo--orreeedd meta-learning for nonlinear systems. In Robotics science and systems, 2021.
Cristian R Rojas, James S Welsh, Graham Goodwin, and Arie Feuer. Robust optimal experiment design for system identification. Automatica, 43(6):993-1008, 2007.
12

---

Published as a conference paper at ICLR 2024
Cristian R Rojas, Tuan-Carlos Aguero, James S Welsh, Graham C Goodwin, and Arie Feuer. Ro- bustness in experiment design. IEEE Transactions on Automatic Control, 57(4))860- 874, 2011.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In CoRL, 2022.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv: /61 1. 1.04201 2016.
Thomas B Schön, Adrian Wills, and Brett Ninness. System identification of nonlinear state-space models. Automatica, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs/1707.06347, 2017a. URL. http:://axxii..rrr///
1707 06347
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b.
Pranav Shyam, Wojeiech Jaskowski, and Faustino Gomez. Model-based active exploration. In ICML, 2019.
Rohan Sinha, James Harrison, Spencer M Richards, and Marco Pavone. Adaptive robust model pre- dictive control with matched and unmatched uncertainty. In 2022 American Control Conference (ACC), 2022.
Laura Smith, Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 International Conference on Robotics and Automation (ICRA), pp. 1593-1599, IEEE, 2022.
Torsten Söderström and Petre Stoica. System identification. Prentice-Hal Hall International, 1989.
Bingjie Tang, Michael A Lin, Iretiayo Akinola, Ankur Handa, Gaurav S Sukhatme, Fabio Ramos, Dieter Fox, and Yashraj Narang. Industreal: Transferring contact-rich assembly tasks from simu- lation to reality. arXiv preprint arXiv:2305.1777 2023.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, 2012.
Marcel Torne Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-ttoreall approach for robust ma- nipulation. Arxiv, 2024.
Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
Andrew Wagenmaker and Kevin Jamieson. Active learning for identification of linear dynamical systems. In Conference on Learning Theory, pp. 3487-3582. PMLR, 2020.
Andrew Wagenmaker, Guanya Shi, and Kevin Jamieson. Optimal exploration for model- rl in nonlinear systems. arXiv preprint arXiv:2306.092 2023.
Andrew J Wagenmaker, Max Simchowitz, and Kevin Jamieson. Task-optimal exploration in linear dynamical systems. In International Conference Machine Learning, pp. 10641-10652. PMLR, 2021.
Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas J Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In ECCV, 2022.
13

---

Published as a conference paper at ICLR 2024
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE international conference robotics and automation (ICRA), pp. 1714-1721. IEEE, 2017.
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B Tenenbaum, and Shuran Song. Densephysnet: Learn- ing dense physical object representations via multi-step dynamic interactions. In RSS, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Chuning Zhu, Max Simchowitz, Siri Gadipudi, and Abhishek Gupta. Repo: Resilient model-based reinforcement learning by regularizing posterior predictability. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL/httss:/ http://///ppeeeeeeeeeeeeeeeeeeee
Id=0IJ3VXDy6s
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 Inter- national Conference on Robotics and Automation (ICRA), pp. 3651-3657. IEEE, 2019.
Shaojun Zhu, Andrew Kimmel, Kostas Bekris, and Abdeslam Boularias. Fast model identification via physics engines for data-efficient policy search. In ICCAI, 2018.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 TASK DETAILS
A.1.1 SPHERE MANIPULATION
In the sphere manipulation tasks, the observation space consists of endeffector position, sphere position, and robot joint angles. During the training of the exploration policy, randomize the location of sphere with = 0.03 to be between x E [0.430.65]. y E [-0.2, 0.23] for training and x: [0.55, 0.65], y [-0.2, 0.23] for evaluation. Parameter ranges for training
1) friction E [1e - 3, 5e - 3] and 2) patch friction ,, 01 02 E [le - 5, le - 3/ and set fixed sphere and goal locations well parameters for the downstream task. For we attach paddle and limit the endeffector position such that the robot has to strike the sphere to the goal.
A.1.2 ARTICULATION
In the case of the articulation environment, the parameter is binary and indicates whether articulation present not. During evaluation, articulation is always turned on and success indicated by 5,3 > le The initial laptop state is randomized over position x E [0.45, 0.65]. E [-0.1, yow E [0.00, 3.14] and lid angle 3 [1.2, 2.5]. The observation space contains the endeffector position, joint angles, position, orientation, and joint angle of the laptop.
A.1.3 ROD BALANCING
Similar to the sphere manipulation, we restrict the exploration policy to endeffector control with òx, dy and attach peg the Franka. The rod has dimensions 0.04 x 0.3 x 0.04 and initializes as & E [0.5, 0.6).y y E [-0.2, 0.2). yow E [0.00, 3.14)
15

---

Published as a conference paper at ICLR 2024
A.2 QUALITATIVE RESULTS
A.2.1 EXPLORATION STRATEGIES LEARNED BY THE AGENT
We evaluate the exploration behavior qualitatively across multiple environments to understand whether the exploration behavior meaningful. In the sphere environment, we observe the agent hitting the sphere multiple times when it bounces off the walls stays within reach. The other environments also experience emergent behavior, as the agent executes a horizontal motion towards the top of the lid if the laptop is mostly open while almost closed laptop causes it to push from top to bottom instead. Finally, when determining the rod's inertia, the policy pushes both sides to gain maximum information about the center of mass through the rotation motions in both directions. See Figure 7 for a visualization of the exploration for the sphere environment.
0.8
0.50
0.52
68
06
0.48
0.38
(a)
(d)
Figure 7: Absolute sphere displacement for different sphere starting locations. Zero means the sphere didn't get hit, higher numbers denote larger displacements. Initial endeffector position marked in red a) Random Coverage, b) PPO Coverage. c) Fisher Coverage, d) Legend.
A.2.2 RECONSTRUCTING ARTICULATION FROM INTERACTION DATA
Figure 8: Articulated object: before (left) and after (middle) exploration with ASID and part-level recon- struction with Ditto (Jian et al. 2022): articulated and static part (right).
A.3 COMPARISON TO NODEL-BASED APPROACHES
A.3.1
ENVIRONMENT SETUP
Figure 9: Environment with multiple spheres. The single red sphere subject to changing friction values while the three blue spheres act distractors.
16

---

Published as a conference paper at ICLR 2024
A.3.2 MODEL-BASED EXPLORATION
In this section, Figure 10 rithm (Shyam et al. 2019)
we compare the performance of our algorithm to that of the MAX algo-
MAX aims to cover the state space through multiple policies throughout training. Since it uses the disagreement between fully learned dynamics models, it gets distracted by novel states induced by the movement of all spheres (red and blue). In contrast, our method based on the Fisher information yields a single policy that seeks out the sphere affected by the changing physics parameters (red) and ignores the irrelevant spheres (blue) even if they lead to novel states.
0.9
0.3
077
077
0.5
210
0.5
2301
WWWNNN
2200
0.5
0.5
0.1
2100
0.1
0.8
2100
0.8
0.2
(a) MAX (checkpoints throughout training)
(b) MAX (last checkpoints)
0.1
COCO
0.1
-0.4
-0.2
0.4
0.2
2000
(c) Fisher information (checkpoints throughout train- ing)
(d) Fisher information (last checkpoint)
Figure 10: Exploration MAX vs. Fisher information: Trajectories collected (30) from policy checkpoints throughout training from the last policy checkpoint. While Model-based Active Exploration (MAX) (Shyam et al. 2019) explores the state space over the course of training, getting distracted by the novel states induced by the blue spheres (c.f. fig. TOap, Fisher information- based exploration (ours) shows directed exploration (c.f. fig. [Od). moving towards the sphere with changing parameters (red Jand yielding single exploration policy.
17

---

Published as a conference paper at ICLR 2024
A.3.3 LEARNED STATE-TRANSITIE MODELS VS. SIMULATORS
We next illustrate the improvement using a simulator vs fully learned dynamics model can give. We train a forward dynamics model (three layer MLP) on data generated both from MAX and our exploration procedure. When evaluated on out-of-distribution trajectories, i.e. trajectories not included in the training data, we find the model to be extremely inaccurate (see Figure TTI. While the simulator extrapolates to unseen states and correctly predicts the movement of the sphere, the model hallucinates movement even when the end-effector does not interact with it at all! These findings make ASID preferable to purely model-based approach.
0.5
0.5
schere Imode)) sphere [ul)
sphame (mretel)) sptaeee (cm)
037
0.7
0.5
0.5
0.2
03-
0.4
0.4
0.2
MAX (pushing actions)
(b) MAX (zero actions)
seluta (nnuid)) sptere
red spfore (mudel)) sphare
-0.4
-0.2
12.4
-0.
-0.2
0.0
(c) Fisher information (pushing actions)
(d) Fisher information (zero actions)
Evaluation a forward model trained on trajectories (30) from MAX (Shyam et al. 2019)) (checkpoints throughout training) and Fisher information (last checkpoint). In contrast our simulator, the learned model fails to predict the red sphere's trajectory accurately even under no contact scenarios (c.f. fig. Ilb fig.
Figure 11: Learned models vs. simulator:
lld).
18

---

Published as a conference paper at ICLR 2024
A.4 POLICY GENERALIZATION
We evaluate the generalization capabilities of our exploration policy for different sphere locations and parameter combinations seen and unseen during training (Figure 13). Since train the policies using an arena setup, We remove the arena during this evaluation to be able to query sphere locations outside of it (Figure
Figure 12: Environment with free space. The red sphere is subject to changing parameter values and position is randomized.
068
40
00
051
053
0/1
62
-22
-035
-0.25
-0.15
-0.05
0.26
2.15
0.25
0:
-035
-0.25
-0.15
05.05
0.26
3.15
0.25
0
traiiing
Eraniig crea
(a) Changing friction parameters (in distribution)
(b) Changing mass parameter (out of distribution)
Figure 13: Generalization capabilities an exploration policy trained on sphere friction for dif- ferent sphere starting locations. The plot shows the success of pushing the sphere spawned the corresponding x-y-location and evaluated over 5 different seeds, i.e., different physics parameter values. Initial endeffector position marked in red Boxes denote sphere locations seen during training (purple box and reachable area of the endeffector (orange box )) Our policies experi- ence surprising generalization capabilities unseen sphere locations as long as they can be reached by the robot (cf.fig. I3a). Furthermore, the actions extrapolate to unseen physics parameters as long as they can uncovered with similar interactions, e-g., pushing (cf.fig. [3b). This is the case be- cause the policy does not have any information about the underlying physics parameters until it hits the sphere.
19


Reviews:
[{"review_id": "zdrr3ze3fk", "reviewer": "Reviewer_WUcQ", "summary": "The paper proposes a system to learn RL policies in simulation which have a high chance of directly trnferring to reality. This is adhieved in a two step process, each step performing RL but with different goals. The goal of the first RL agent is to learn an exploration policy that can collect meaningful simulator calibration data from a single run in the real world. The second RL step learns to achieve the desired goal by learning in a simulator that got calibrated using the once real world run. The main contribution is the first step which uses Fisher information, widely used in system identification, as the cost function of the RL agent.", "questions": "- What is the runtime of the entire system?\n- How hard is it to come up with a simulation for the first goal?\n- How precise does the simulator have to be?\n- There are simplifying assumptions made for the exploration Fisher loss, how limiting are they?\n- Equation 4 states that an initial distribution of parameters is assumed, how is this obtained?\n- The text states that the system isn't using a differentiable physics engine. If one was used, what would this mean for the method?\n- How many parameters can be estimated and what happens when parameters are coupled or jointly multi-modal?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Treating simulator calibration as system ID is an interesting way to approach things.\n- The modifications of the Fisher information to make it suitable for RL training is also nice.\n- Paper is well written and easy to understand and follow.\n- Outline of questions to answer in the experiments section is a good addition.", "weaknesses": "Not per se a weakness of the method but the expectation set by the beginning of the paper. There two aspects that make RL challenging to deploy on real system is safety and sample efficiency. The writing gives the impression that the paper tackles both aspects, when it really tackles the sample efficiency aspect. There is no guarantee that the exploration is safe only that it should be more informative. The proposed approach is interesting as it is and I don't think not dealing with safety aspects is an issue.\n\nAnother aspect that does not really fit with the paper is the geometric learning aspect. It does not integrate well with the rest of the paper. The proposed approach is also highly specific and not generally usable. For example, the shape reconstruction is not going to work for complicated objects and will not result in accurate physical simulation outcomes. It is interesting that something like this can be done, but way it is presented and the amount of space available to that aspect makes it hard to fully understand and makes the results sound rather underwhelming.\n\nOne aspect that it unclear from the paper is how specific the resulting exploration and task policies are. How generalizable of a policy does the system learn at the end of the day? For example, does the ball pushing policy work only for the specific environment with that breakdown of friction patches and coefficients or is the policy more general and can be used to push balls in a variety of environments? Put differently, do I need to learn a new task policy and calibrate the simulator for every minor varioation of the task description?\n\nThe work mentions that it assumes the optimal policy can be found. That is a rather big assumption for RL as finding the optimum is not guaranteed and the other aspect is that often the reward function does not truly represent what we want to optimize for. Does the proposed approach actually need to find the optimum or is a \"good enough\" policy also acceptable?\n\nOverall the experimental results are nicely presented and show good performance. Two things that could be improved are the discussion of the outcomes. There is little information about failure modes and their explanation, for example. The other part is that Section 5.3. makes sense under the hypothesis that good exploration coverage leads to good RL task performance. Is it possible to show this more directly in that section?\n\nAs side comment, maybe using \\Pi_{task} for the learned task policy, to mimic \\Pi_{exp}, could be a nice way to make it even clearer that there are multiple policies and what their goals are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AFv4wpOSqs", "reviewer": "Reviewer_gS1p", "summary": "This paper presents a method for active exploration for model based reinforcement learning in the context of robotic manipulation. The paper introduces an exploration policy based on the Fisher information matrix of the parameters of the model. Then, they also include a vision system for scene reconstruction and experimental evaluation based on a robotic manipulator, both in real and simulation environments.", "questions": "-I do not fully understand the reference to REPS as that is a model-free RL method. There is no transition model estimation.\n-It seems that the system relies on the assumption that a learned simulator is able to generate accurate trajectories, but that is not the case for out of distribution trajectories. I understand that exploration precisely minimizes that effect, but the probabilistic model should be able to capture the lack of information in out of distribution data. Currently, the only uncertainty comes from the noise if I understand correctly.\n-The scene reconstruction part seems to be a part of the specific experiments presented in the paper, but it is unrelated to the exploration pipeline.\n-How did you use RANSAC for tracking?\n*****\nPost discussion update:\nIf I understood correctly, your method is actually similar to MAX, but instead of using a statistical model as many methods, yours is a physically-informed model, but a parametric model nonetheless. I can see the benefit of using a physically-informed model in a robotics setup. Clearly it is an advantage. However, when evaluating this kind of setups, one has to evaluate the scenario where there are mismodelling errors. For example, most robot models asume rigid-body dynamics, while real life dynamics in high acceleration/forces scenarios suffer from elastic behaviors and therefore are non-Markovian. If the setup is robust, as you said, the policy should be useful (even if suboptimal), but you have to show robustness to mismodeling errors that can make the solution diverge from the actual dynamics.\n\nAlso, because you are not learning any policy in section 4.2.1, I wouldn't say that you are using REPS. Instead, if I understood correctly, you are doing supervised learning using natural gradients (which also includes the KL bound). In fact, when you want to do trajectory matching, such as in apprenticeship learning and inverse reinforcement learning, the least-squares loss is problematic, and previous work actually tries to minimize the KL divergence between tau_sim and tau_real directly (see for example: Boularias, Abdeslam, Jens Kober, and Jan Peters. \"Relative entropy inverse reinforcement learning.\" Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011.)\n\nI agree that the default HalfCheetah does not have variable dynamics, but it can be easily modified (for example, change weight or link length) as has been previously done in other works. For example: https://arxiv.org/pdf/1810.03779.pdf (includes code for some Gym envs).\n\nThis is maybe just me being pedantic, but frame/point cloud detection is not tracking. Tracking requires some sequential estimation. Note that this is not a critique on the section: continuous detection might be enough for the experiments. However, as before with the model-based RL it is a suggestion on proper naming conventions to clarify the text.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The main strength of this paper is the fact that part of the experiments are done in a real manipulator. Also, the pipeline of doing active exploration for model learning (system identification) is fundamental for robotic applications.", "weaknesses": "The main weakness for this paper is that this pipeline is very similar to other exploration methods model-based RL. For example:\nShyam P, Jaśkowski W, Gomez F. Model-based active exploration. In International conference on machine learning 2019 May 24 (pp. 5779-5788). \n\nPathak D, Gandhi D, Gupta A. Self-supervised exploration via disagreement. In International conference on machine learning 2019 May 24 (pp. 5062-5071). \n\nIn fact, the pipeline is quite similar to Shyam et al. albeit the metrics and models used are different. However, due to the similarities in the process, those papers should be discussed and, ideally, included in the comparison.\n\nWhile the experimental section is one of the strengths due to the evaluation in a realistic robotic scenario, the methods should also be evaluated on standard benchmarks for comparison, such as HalfCheetah. The baseline used [Kumar2019] seems very weak (in Fig 4 it does not explore at all). Furthermore, the work of Kumar2019 does not seem to be related to exploration with mutual information as stated in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TdQ4pyeRjj", "reviewer": "Reviewer_zEL2", "summary": "This work lays out a framework for robotic manipulation systems to explore and model unknown environments, as well as train a policy to succeed at control tasks within this environment. This generic pipeline for sim to real transfer is called Active Exploration for System Identification, or ASID, and it involves three stages: exploration to gather information about the environment, refinement of this simulation with the data, and training a policy in the learned environment. This approach is shown to be both highly successful and very data efficient, with real work robotics examples shown both in simulation and on real hardware.", "questions": "Potential typos: \n1. End of section 1 says \"signal episode\", should this be \"single episode\"?\n2. Section 4.3 says \"zero-short\", should this be \"zero-shot\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and has a nice flow to it. Organization and structure both help with this as well. \n2. The sections on related work and preliminaries do a good job of giving the appropriate context/notation. \n3. The tasks chosen to demonstrate this approach were challenging, informative, and speak to the efficacy of the approach. \n4. Hardware experiments look convincing. \n5. The connections to A-optimal experiment design are insightful and appropriate.", "weaknesses": "1. More detail on why the Fisher Information is used vs other methods (observability Grammian, Kalman Filter covariance).\n2. The numbers in the heatmaps in Figure 4 are hard to read, maybe block font for the numbers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jjG8L8lSX3", "reviewer": "Reviewer_4bkX", "summary": "This paper proposes a framework for model-based RL aimed at learning model parameters as well as the optimal policy given the model. The method seeks to address to the sim-to-real gap by proposing an efficient policy for exploring the environment inasmuch as that exploration improves the model. At each step, the method finds the policy that approximately maximizes the Fisher Information in the trajectories we expect the policy to encounter when rolled out. The method is experimentally validated on a number of real world environments.", "questions": "1. In Section 4.2.1, I was expecting to see the standard SysID loss, which is to maximize the likelihood of the data (in this case trajectories) given model parameters. You find the distribution that maximizes likelihood for domain randomization. It seems to me that without some sort of entropy maximization term in the objective, or bootstrap, you would just end up with an MLE objective, whereas it seems like you want to find the Bayesian posterior of models given data. Can you comment on how your objective relates to that of finding a Bayesian posterior?\n2. For a paper proposing active exploration for the sake of system identification, I wanted to see more discussion of the following: a) regret minimization, i.e. can you prove that your method minimizes regret and achieves the best policy with the fewest interactions with the environment? b) identifiability, i.e. can you say anything about whether all system parameters will be uniquely identified with infinite interactions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an important problem, i.e. directing exploration of an environment in an effort to reduce model uncertainty.\n- The paper proposes a seemingly novel approach of finding policies that maximize the Fisher information.\n- The paper validates the approach using real-world experiments", "weaknesses": "- Presentation/clarity can be improved: specifically, the abstract and introduction mostly describe the field of active exploration for system identification and adaptive control, as opposed to the specific method proposed, which appears to overstate the paper’s novelty. Furthermore, the approach warrants a better intuitive explanation. As I understand it, the Fisher Information objective attempts to quantify the sensitivity of model parameters to trajectories expected given some policy. Therefore, maximizing this objective yields a policy that, when executed, yields the maximum additional information about the model parameters.\n- Lack of baselining/adequate discussion of other methods that use the Fisher information objective. The statement “As compared to these works, a primary novelty of our approach is the use of a simulator to learn effective exploration policies” seems too strong and overstated given that there are entire fields dedicated to this, and “the application of our method to modern, real-world robotics tasks” is an inadequate claim to novelty.\n- Literature review can be improved with a discussion of the following:\n    - Bayesian RL/Bayes-adaptive MDPs: \tM. Duff. Optimal Learning: Computational Procedure for Bayes-Adaptive Markov Decision Processes.  PhD thesis, University of Massachusetts, Amherst, USA, 2002. \n    - PILCO:\n        - Deisenroth, Marc, and Carl E. Rasmussen. \"PILCO: A model-based and data-efficient approach to policy search.\" Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.\n    - Adaptive MPC:\n        - S. M. Richards, N. Azizan, J.-J. Slotine, and M. Pavone. Adaptive-control-oriented meta-learning for nonlinear systems. In Robotics: Science and Systems, 2021. URL https://arxiv.org/abs/2204.06716.\n        - Sinha, Rohan, et al. \"Adaptive robust model predictive control with matched and unmatched uncertainty.\" 2022 American Control Conference (ACC). IEEE, 2022.\n    - System identification in partially observable environments:\n        - Menda, Kunal, et al. \"Scalable identification of partially observed systems with certainty-equivalent EM.\" International Conference on Machine Learning. PMLR, 2020\n        - Schön, Thomas B., Adrian Wills, and Brett Ninness. \"System identification of nonlinear state-space models.\" Automatica 47.1 (2011): 39-49.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)