AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer
Authors: Allen Z. Ren
Hongkai Dai
Benjamin Burchfiel
Anirudha Majumdar
Venue: CoRL 2023 Poster (2023)
Keywords: Contact-rich manipulation
sim-to-real transfer

Abstract: Simulation parameter settings such as contact models and object geometry approximations are critical to training robust manipulation policies capable of transferring from simulation to real-world deployment. There is often an irreducible gap between simulation and reality: attempting to match the dynamics between simulation and reality may be infeasible and may not lead to policies that perform well in reality for a specific task. We propose AdaptSim, a new task-driven adaptation framework for sim-to-real transfer that aims to optimize task performance in target (real) environments. First, we meta-learn an adaptation policy in simulation using reinforcement learning for adjusting the simulation parameter distribution based on the current policy's performance in a target environment. We then perform iterative real-world adaptation by inferring new simulation parameter distributions for policy training. Our extensive simulation and hardware experiments demonstrate AdaptSim achieving 1-3x asymptotic performance and 2x real data efficiency when adapting to different environments, compared to methods based on Sys-ID and directly training the task policy in target environments.

AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer
Allen Z. Ren', Hongkai Dai Benjamin Burchfiel", Anirudha Majumdar' PPiineeoon University, Toyota Research Institute
Abstract: Simulation parameter settings such as contact models and object geometry ap- proximations are critical to training robust manipulation policies capable of transferring from simulation to real-world deployment. There is often an irreducible gap between simulation and reality: attempting to match the dynamics between simulation and reality may be infeasible and may not lead policies that perform well in reality for specific task. We propose AdaptSim, a new task- adaptation framework for sim-to-reat transfer that aims to optimize task performance in target (real) environments. First, we meta-leamy an adaptation policy in simulation using reinforvement learning for adjusting the simulation parameter distribution based on the current policy's 's performance in a target environment. We then perform iterative real-world adaptation by inferring new simulation parameter distributions for policy training. Our extensive simulation and hardware experiments demonstrate AdaptSim achieving 1-3x asymptotic performance and ~2 real data efficiency when adapting to different environments, compared methods based on Sys-ID and directly training the task policy in target environments.
Introduction
Learning robust and generalizable policies for real-world manipulation tasks typically requires a substantial amount training data. Since using real data exclusively can be very expensive or even infeasible, we often resort to training mostly in simulation. This raises the question: how should we specify simulation parame- ters maximize performance in the real world while minimizing the amount of real-world data we require?
A popular method is to perform domain randomization [1. 2, 3, 4]: train a policy using a wide range of different simulation parameters the hope that the policy can thus han- dle possible real-world variations dynamics or observations. However, the trained policy may achieve good average per- formance, but perform poorly a particular real environment. There has been work in performing system identification (Sys-ID) for providing point a distributional estimate of parameters that best matches the robot or environment dy- namics exhibited real-world data. This estimation can be performed using either a single iteration [5] or multiple ones [6]. These adaptive domain randomization techniques allow
Beration
Iteraionn
Itaration3
training policies suited to specific target environments.
Figure AdaptSim iteratively improves task performance in dynamic scooping task under "irreducible" sim-to-real gap.
While simple objects such as a box and its properties like the inertia can be well-modeled, there is substantial amount of "irreducible" sim-to-real gap in many settings such as contact-rich manipulation tasks. Consider the task of using a cooking spatula to dynamically scoop up small pieces food from a table (Fig. 1). The exact geometry of the pieces and spatula is difficult specify, deformations such the spatula bending against the table are not yet maturely implemented in simulators, and contact models such as point contact have been known poorly approximate the complex real- world contact behavior [7] in these settings. In this
'Webpage:
github.ion. itoi/i///paaa AddpttStt
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

case, real environments are out-of- of-domain (OOD) from simulation, and performing Sys-ID of simulation parameters might fail to train a useful policy for the real world due to this inherent irreducible gap.
Contributions. In this work we take a task-driven approach: instead of trying to align the simulation with real-world dynamics, we focus on finding simulation parameters such that the resulting policy optimizes task performance. Such an approach can lead policies that achieve high reward in the real world even with an irreducible sim-to-real gap. We consider settings where the robot has access to a simulator between iterations of real-world interactions. allowing it to observe world dynamics and adapt the simulator accordingly with the goal of improving task performance in reality. We propose AdaptSim - a two-phase framework where (i) an adaptation policy that updates simulation parameters is first meta-trained using reinforcement learning in simulation, and (ii) then deployed on the real environment iteratively. Training the adaptation policy to maximize task reward enhances the efficiency of real data usage by identifying only task-relevant simulation parameters and helps trained policies better generalize to OOD (real) environments. We demonstrate our approach achieving 1-3x asymptotic performance and ~2x real data efficiency in OOD environments in three robotic tasks including two that involve contact-rich manipulation, compared to methods based on Sys-ID and directly training the task policy in target environments.
2 Related Work
Sim-to-real transfer in robotics has been primarily addressed using Domain Randomization (DR) techniques [1, 8, 9, 10, 11, 12, 13] that inject noise simulation parameters related to visuals, dynamics, and actuations. Below we summarize techniques that better adapt to real environments.
Sys-ID domain adaptation. Inspired by classical work in Sys-ID [14, 15), there has been popular line of work identifying simulation parameters that match the robot and environment dynamics in the real environment. BayesSim [6] and follow-up work [16, 17] apply Bayesian inference to iteratively search for a posterior distribution of the simulation parameters based on simulation and real-world trajectories. However, these methods consider relatively well-modeled environment parameterizations such as object mass or friction coefficient during planar contact; Sys-ID approaches can be brittle when the simulation does not closely approximate the real world [13, 18].
Task-driven domain adaptation. AdaptSim better fits within a different line of work that aims to find simulation parameters that maximize the task reward in target environments. Muratore et al. [19] apply Bayesian Optimization (BO) to optimize parameters such as pendulum pole mass and joint damping coefficient real pendulum swing-up up task. Other work focus on adapting to simulated domains only [20, 21, 22]. One major drawback of these methods is that they require large number of rollouts in target environments (e.g., 150 in [19]), which is very time-consuning for many tasks requiring human reset. AdaptSim meta-learns adaptation strategies in simulation and requires only few real rollouts for inference (e.g.. 20 in our pushing experiments).
3 Problem Formulation
Environment. In simulation, we consider space Ω that parameterizes quantities such as friction coeffi- cients and dimensions of geometric primitives. Let denote a distribution of sim parameters with support on Ω. Denote single sim environment E and real environment E".
Task Policy and Trajectory. We denote task policy E ET:O~+A A that maps the robot's observation or action a, Running it in an environment results in a state-action trajectory T(T;E):: [0.7) x x S A with time horizon T. The trajectory is also subject to an initial state distribution. We specify tasks for the robot using a reward function (e.g,, pushing some object to a specific location on the table), and let R(() E [0,1] denote the normalized cumulative reward accrued by a trajectory. We let R(::E) denote the reward of running the task policy æ the environment E, in expectation over the initial state distribution.
Goal Our eventual goal is to find task policy that maximizes the task performance in real environment E,. Instead of directly searching for the policy, we search for the best sim parameter distribution E for training ı in the following levee optimization objective:
sup R(Tē:E"), where πÊ. sup J [R(T;))}, E
(I)
8
2

---

Phase 1: Meta-Learning Adaptation in Sim
Phase 2: Iterative Sim-to- Real Adaptation
aararwter dissibuttoo
Collecting adaptation experiences
23n target
Training adaptation pollcy maximize target
Adaysiiee
anvironment
Sin paiateser space
Adaptation step
Task pollcy training in sim with &
Evaluation at target (sim or real +)
Sim parameter
update
Figure 2: AdaptSim consists of phases: (1) meta-training adaptation policy in sim by maximizing task reward on randomly sampled simulated target environments; (2) iteratively adapting simulation parameter distributions based on real traeectories. The upper-right illustration shows that using only a few real trajectories, the task policy adapted push the bottle closer the target location (yellow cross).
the optimal task policy for E. Performing the outer level of (I) requires interactions with ET (the real world); we allow a small budget of such interactions. We emphasize that the objective above identifies the optimal distribution of simulation parameters for maximizing task performance, unlike objectives that attempt to match the dynamics between simulation and reality.
Approach
One way to solve (1) is to perform blackbox optimization & by evaluating (() [19], which requires large budget of real trajectories (see results in Sec. 6). AdaptSim instead amortizes the expensive outer loop to simulation: it solves (I) for many simulated environments, learns the mapping to the solutions, and then infers the solution for E,.. There are two phases (Fig. 2):
1) Meta-learn the adaptation policy in sim: randomly sample target environments E El in sim, and then train an "adaptation" policy (E using RL to maximize task reward in E", by updating the sim parameter distribution (and the corresponding task policies) iterations.
Iteratively adapt sim parameters with real data: given a real environment E", iteratively infer better sim parameter distributions using the trained and few real trajectories; the task policy is iteratively fine-tuned in sim to improve task reward with the updated parameter distribution.
4.1 Phase 1: meta-learning the adaptation policy in sim
In order to correctly infer simulation parameters for unseen real environment test time, first train the adaptation policy to infer better parameters for many simulated target environments. This phase happens entirely in simulation. Formally, we model the problem as partially-observable contextual bandit [23].
Definition 1 A Simulation-Adappatii Contextual Bandit (SA-CB) is specified by tuple (Q,,TP,RR)) P.R):
Ωi is the space of contexts. Each context corresponds simulated target environment E*; the context not directly observable.
T is the space of partial observations of the context. Each observation corresponds to trajectory observed by running the task policy a given context.
P is the space of actions. An action corresponds to choosing a sim parameter distribution
is the reward associated with choosing an action a particular context (i.e., the reward R(x of the task policy T& trained with E when deployed in the target environment E*).
may difficult to infer the optimal E E P using a single iteration of interactions with the target environment the current task policy fails badly in the target environment, the interaction may reveal little information. Thus, we iteratively apply incremental changes to E, with the parameter distribution initialized as E. Solving the SA-CB (using techniques that detail below), meta-leam an adaptation

---

policy f(E,T) to maximize:
E Bt-dda Eo--dp
E
7 R( F
1=0
where E1+1 = E, DE, ((C.
and Up and 1/p are uniform distributions over S2 and P respectively, and 7 is the discount factor. This the expected discounted sum of task rewards over multiple interactions from i i= the adaptation horizon over random sampling of simulated target environment and initial sim parameter distribution.
Sim parameter distribution space. We choose the space P of possible simulation parameter distri- butions to be Gaussian with mean bounded within and a fixed variance. We also use fixed step size for adapting each simulation parameter. rang- ing from 10% to 15% of the parameter range de- pending on the dimension of Ω thus the set of possible Ae along each dimension is {6,-5.0).
(a-1)
(a-2)
Figure 3: Task-polic trajectories better reveal task-relevant information such as scooping dynamics under fast contact.
Algorithm Meta- Mlea-learning the adaptation pol- icy in sim
Task fask-y trajectory as observation. We have chosen the task policy π² to generate the trajectory observations used by the adaptation policy. Our intuition is that, compared arbitrary policies or ones that generate the most "informative" trajectories in terms of dynamics [24], g better reveal the task-relevant information of the target environment. In the scooping task, the robot needs to attempt to scoop up the pieces so it can learn about the effect of the environment on the task (e.g., piece with a flat bottom is generally harder to scoop). Simply pushing the pieces around does not exhibit the behavior of the pieces under fast contact (Fig. 3).
Require: (R),,,,,R), SA-CB
Require: 00 replay buffer
Require: Se Ø, set of all simulation parameter
distributions (and their task policies) used
1: Initialize c
2: for to K do Sample target E" ~226 and u4=
4: for i+0 / do
5:
Train task policy मē. (Sec.A2.2)
6:
Collect ( :E") and R(At :E*)
7:
Sample random De, infer Ae,
((E.,
Training the adaptation policy using RL. The adap- tation policy f is parameterized using a Branching Du- eling Q-Network [25], which outputs the state-action value of choosing any of the {6, (6,0 along each action dimension. It takes in (1) the vector of the mean of cur- rent simulation parameter distribution and (2) trajectory observation. We apply reinforcement learning (RL) to train f to maximize Eq. (2). In simulation, we collect K "adaptation trajectories"" each trajectory is set {EE, Ae,, H( E"", ( )))) and saved in replay buffer Sy. Since each step involves training the corresponding task policy ~े which can be expensive, we apply off-policy Double Q-Learning [26] for sample efficiency. Using this, the adaptation policy outputs the greedy action of parameterized function, f(E,r) argmax (E) We use c-greedy exploration with initialized 1 and annealed O.
8:
Update 1 E
9:
Add (T.)) to Sy
10:
Add & (and nē.) Ss
11:
end for
12: Train f using Double Q-I.carnng and Sy
13:
Anneal towards 0
14:
end for
15:
return Se
This constitutes the first phase of AdaptSim. Algorithm details the steps for collecting adaptation trajectories in the inner loop (Line 4-15) and meta-learning the adaptation policy. We save all distributions (and their corresponding task policies, omitted in notations for convenience) in a set Sg, which are used again in the second phase. Training the task policy for each E is the most computationally heavy component of Algorithm 1; in Sec. A2.2 explain the heuristics applied to allow re-using task policies between & in order to improve computational efficiency.
4.2 Phase 2: iteratively adapt sim parameters with real data
After meta-training the adaptation policy to find good task policies for diverse set of target environments in simulation, we can apply it for inference and perform adaptation for the real environment E". Algorithm details the iterative process. We apply the same adaptation process as inner loop of Algorithm for r" iterations: train the task policy in simulation, evaluate it in the real environment, and infer the change of simulation parameters based real trajectories. We always apply the greedy action from f(E,T) (e=0).

---

ap-ainned bottes for the pushing
Henry Uight
srrdatted
Necpmene
patch
patch
Pushing task, Sim Pushing
Scooping task, Sim Scooping
for
Figure 4: Setup of the dynamic pushing and dynamic scooping tasks in both simulation and reality.
Since we have sampled large set Sg of parameter dis- tributions and trained their task policies in Phase 1, may re-use them here. At the beginning of Phase 2, we sample Sy set of N distributions saved in Sg, as the initial distributions to be adapted independently. Usually we pick N =2 considering the trade-off between num- ber of real trajectories needed and convergence task performance (see Appendix A5 for analysis).
Algorithm 2 Iteratively adapt sim parameters with real data
Require: E", real environment
Require: (PL),,P,I)), SA-CB
Require: adapttati policy trained in Phase
Require: Sy, set of sim parameter distributions (and
corresponding task policies) from Phase
1: Sample Sy from Sy
2: for (4-i4-000 |" do
& E
5 Tasks
Train or fine-tune the task policy nद, in
sim
Next we detail the three robotic tasks for evaluating AdaptSim and baselines. We choose these tasks and design the environments highlight the irreducille gap between training and test domains.
5:
Collect T(TT;; F") and R(Tē,; F5)) in real
6:
Update E41 +- (()
7: end for
8: end for
5.1 Swing-up of lincarized double pendulum
return मड. with highest R(=b²)
This is classic control task where the goal is swing up simple double pendulum with two actuated joints at one end of the two links. We consider the dynamics lincarized around the state with the pendulum at the top, and thus the optimal policy can be solved exactly using Linear Quadratic Regulator (LQR) [27] for particular set of simulation parameters (i.e., Dirac delta distribution). The task cost (reward) function defined with the standard quadratic state error and actuation penalty. The trajectory observation is evenly spaced points along trajectory of the two joints.
Simulation setup. The environment is parameterized with four parameters: my and m2 E [1,2], point of the two joints, and 6j and by [1.2]. damping coefficients of the two joints. The dynamics is simulated with numerical integration without a dedicated physics simulator.
5.2 Dynamic table-top pushing of bottle
The robot needs to dynamically push a bottle to a particular target location on the table (Fig. 4). Since the target can be outside the workspace of the robot, the robot must push objects with high velocity - causing them slide after short period of contact. The task policy is parameterized with a neural network that maps the desired target location to action including (1) planar pushing angle and (2) robot end-effector speed (sec Appendix A4 for visualization) and the predicted reward. The network then acts state-value (Q) function and is trained off-policy while simulated trajectories are saved in replay buffer. The task cost (reward) is defined as the distance between the target location and the final location of the bottle. The trajectory observation is either (1) the final 2D position of the bottle only, or (2) evenly spaced points along the 2D trajectory- we consider both representations the experiments.
Simulation setup. We employ the Drake physics sim- ulator [28] for its accurate contact mechanics. In this simulated enviromment, a small patch of the table is sim- ulated with different physics properties, simulating a wet or sticky area on the work surface. Parameter settings
Notation
Description
Range
table friction coefficient 00.05.0.2
hydroelastic modulus [7]
11e1,166
patch friction coefficient [0.20,0.80]
3p
patch lateral location
[00.10,0.10]
for this task are shown in Table 1. The hydroelastic mod-
Table Sim setup pushing task.
ulus is parameter of the hydroelastic contact model [7]
implemented in Drake it roughly simulates how "soft" the contact is between the objects, with lower values being softer.
5

---

Real setup. Two 3D sD-printed bottles (Fig. 4, Heavy and Light) with the same dimensions but different materials and masses are used. With an idealized model, the sliding distance should only depend on the contact surface but not the mass - which is the case in simulation - but in real experiments, we find the two bottles consistently travel different distances. Additionally, Heavy tends to rotate slightly despite being pushed straight due slightly uneven bottom surface. This type of unmodeled effect exemplifies the irreducible sim-to-real gap. We also adhere a small piece of high-friction Neoprene rubber to the table, which decelerates the bottle and further complicate the task dynamics.
5.3 Dynamic scooping of food pieces with a spatula
The robot needs to use a cooking spatula to scoop up small food pieces on the table (Fig. 4). It is a challenging task that requires intricate planning of the scooping trajectory - we notice humans cannot complete the task consistently without a few trials to practice. The task policy is parameterized with a neural network that maps the initial positions of the food pieces to parameterization of the scooping trajectory: (1) initial distance of the spatula from the pieces, (2) initial pitch angle of the spatula from the table, and (3) the timestep to lift up the spatula (see Appendix A4 for details), and the predicted reward. The task reward is defined as the ratio of food pieces on the spatula at the end of the action. The trajectory observation is evenly spaced points along 2D trajectories of the food pieces.
Simulation setup. We again use the Drake simulator: The parameter settings are shown in Table 2.
Real setup. Six different kinds of food pieces are used (Fig. 4) (1) chocolate raisin, (2) (fake, rubber-like) sliced carrot, (3) (fake, rigid) sliced cucumber, (4) raw Brussels sprout, (5) raw sliced mushroom, and (6) Oreo cookie. They cover different shapes from being round, ellipsoidal, to roughly cylindrical, and also have different amounts of deformation and friction.
Notation
Description
Range
fr
friction coefficient
00.25,0.4]
e
hydroclastic modulus
114,5555
g
food piece geometry glllisooidcylinddee
h
food piece height [1.5cm, 2.5cm
Table 2: Sim sctup for the scooping task.
Experiments
Through extensive experiments below, we demonstrate that AdaptSim improves asymptotic task perfor- mance compared to Sys-ID and other baselines when adapting to real and OOD simulated environments, while also improving data efficiency. For baselines, first we consider methods that directly optimizes the task policy: (1) Uniform domain randomization (UDR): train a task policy to optimize the average task reward over environments from Ug; (2) UDR+ Targett fine-tune the task policy from UDR with real data; (3) Ieareet directly train a task policy with data the target environment only by fitting a small neural network that maps action to final reward. The policy then outputs the action with the highest predicted reward. With enough real data, this baseline should act the oracle or upper bound of task performance, but can be inefficient. Next, we consider two that perform SysID and iteratively train the task policy like AdaptSim: (4) SysLD-Bayes [6, 29]: iteratively infer the sim parameter distribution based on real trajectories to match dynamics in sim and reality, known as BayesSim; (5) SysID-Point: infer a point estimate of the sim parameter instead of distributional one (we hypothesize that in some settings randomizing sim parameters with a distribution can negatively impact task policy training).
6.1 AdaptSim achieves better task performance through adaptation
Sim-to-Sim Adaptation. We perform experiments for all baselines adapting to different WD (Within- Domain) and OOD simulated environments. WD environments are generated by sampling all simulation parameters within 22 of each task, and OOD environments are generated by sampling some parameters outside Q (see Appendix A5 for details). Table shows the adaptation results in the target environments in the three tasks. While Sys-ID baselines achieve high reward in WD environments, AdaptSim outperforms Sys-II baselines in almost all OOD environments.
Sim-to-Real Adaptation. Next we perform experiments for adapting to real environments. Fig. shows the average reward achieved at each adaptation iteration in the pushing and scooping tasks. Generally the performance gap between AdaptSim and Sys-ID baselines is larger in reality, with AdaptSim achieving better performance. In the scooping task, for example, AdaptSim is able to train a task policy for sliced cucumbers with decent performance (60% success rate); the pieces are very thin and difficult to scoop under (Fig. 8). Other baselines fail to scoop up the pieces.

---

Double Pendulum Swing-Up
Bottle Pushing
Food Scooping
Method
WD
00D-1
OOD-2
OOD-3
00D4
WD
00D-1
0OD-2
0OD-3
00D4
WD
OOD-1
OOD-2
00D-3 OOD-
AdapeSim
0.98
0.96
0.95
0.95
0.98
0.95
0.87
0.73
0.86
0.77
1,00
0.64
1.00
1.00
0.55
SystD-Bayes [6] 0.85
0.76
0.79
0.23
0
0.98
0.80
0.65
0.81
0.79
0.90
0.66
0.81
1.00
0.36
SydDD
0.60
0.78
0.90
1,00
UDR
0.68
0.65
0.61
0.67
0.58
0.65
0.22
0.43
0.55
0.12
UiR+Taage
0.70
0.60
Learla
0.91
0.75
0.66
0.74
0.71
0.03
0.00
0.25
0.26
0.03
Table 3: Sim-to-Sim Adaptation. Best average reward achieved over adaptation borizons at different WD and OOD simulated target environment in the three tasks. For the pendulum task, the values are normalized in [0,1] using the reward achieved by UDR (lower bound) and by using the best possible parameters within ! (upper bound, estimated with exhaustive sampling). For the pushing task, the values normalized with 20cm as the maximum error, which is the range of possible goal locations in the forward direction.
1.0
Heavy bottle
Normalized reward Normalized reward
Chocolate raisin
(Fake) sliced capret
(Fake) sliced cucumber
0.9
0.8
0.6
0.8
0.4
0.7
0.2
AdapiSim
Sys D-Bayes
0.0
Normalits rewer
0.
1.0
Light bottle
Leamintarget
UDR
DysiD-Point
1.0
0.8
Brussels
sprout
Sliced mushroom
Cookie
0.6
0.8
0.4
0.2
0.2
0.6
0.0
Iterations
Iterations
Figure 5: Sim-to-Reat Adaptation. Reward achieved over adaptation iterations by all methods, in the task of pushing (left) and scooping up (right) different real objects (see Fig. 4 for images). Results are averaged over 10 trials in the pushing task and 5 in the scooping task.
Real data budget
6.2 AdaptSim improves real data efficiency Pushing task. We compare AdaptSim with Learn- InTarget and UDR+Target with different number of real data budget. With enough data, LearnIn Tar- get and UDR+Targer should achieve high reward in the target environment. We do not compare with Sys-ID baselines here since Sec. 6.1 shows they typically fail to achieve the same level of task per- formance in real environments. In the task of pushing Heavy bottle, Table shows that AdaptSim achieves similar level of task performance (~0.83) using only 16 trials while LearnInTarger and UDR+Targes uses 40. Fine-tuning with real data in UDR+ Target is ineffective until the real budget is sufficient and negatively impact the performance in the low-data regime (e.g.. 4 and 8). This also exemplifies using simulation to amortize data requirements for policy training. We also introduce a new baseline BayesOpt here based on [19] that directly optimizes Eq. (1) with Bayesian Optimization. However, with 24 rollouts (the minimum needed to initialize the optimization) it only achieves 0.65.
Method
0
4
8
16
24
32
40
48
AdaptSim 0.30 0.69 0.80 0.83
0.84
0.84 0.82 0.83
LeamlnTarget 0.05 0.04 0.63 0.69
0.76
0,80 0.84 0.83
UDR+Target 0.63 0.56 0.62 0.66 0.68 0.74 0.82 0,82 BayesOpt
0.65 0.72 0.79 0.80
Table 4: Adaptation Data Efficiency. Normalized reward achieved using different amount of real data in the pushing task with Hcavy bottle.
Larger improvement in scooping task. While LeamIn'Target and UDR+Target achieve reasonable performance in the pushing task, Leannn Target achieves low reward on all the food pieces in the scooping task, and UDR+ does not improve upon the performance of UDR policies. The action space in the scooping task is more complex and requires significantly more data to search for improve task policies. AdaptSim's 's adaptation pre-training in simulation considerably amortizes the real data requirement.
6.3 AdaptSim finds sim parameters that are different from ones from SysID
We expect that AdaptSim finds simulation settings that achieve better task performance while not necessarily minimizing the full dynamics discrepancies between sim and reality. Fig. shows SysID--Baes finds parameters that are closer the target the parameter space, but for the pendulum task, such parameters lead inferior task reward compared to those found by AdaptSim. Moreover, compute the dynamics discrepancy, measured as the total variations between trajectories in the target environment and in the
7

---

iteraiion
Iteration
iteration
Iteration
Iteration
0.20
Toble friction coefficient
Leb
Hydroslastic modulus
0.30
0.10
0.05
0.00
Patch friction confficient
Patch iateral position (m
0.75
0.50
0.0
Adap:Si
0.25
Sys1C-Uaaes
Iterations
Figure 7: Adaptation in Pushing Task. AdaptSim correctly learss to push the bottle swiftly and close to the target. The task-relevant sim parameters learned by AdaptSim noticeably differ from those by SyslD-Baye which tends underestimate table and patch friction, resulting in less forceful push of the bottle and worse task performance.
Friction coefficient
Hydroelastic modalus
5.0
0.35
1.0
late lift off
Iteration
Food piece geomnetry
piece height (sm).
Arl Bayes
early lift off
Iterationn
Figure 8: Adaptation Scooping Task. With AdaptSim, the cucumber is successfully scooped up by lifting up the spatula off the table late; otherwise, the piece slips off the spatula. AdaptSim infers an ellipsoidal shape (g - 1, food piece geometry), while SysID-Bayes infers a cylindrical shape.
environment with adapted parameters. The results are 17.6 vs. 12.1 for AdaptSim and SysID- SysID-Bayes Bayes in OOD-1 environment, 21.7 and 1.1 in OOD-2, 39.9 and 16.4 in OOD-3, 75.8 and 56.4 in OOD-4. Thus for all four OOD target environments, SysID-Bayes finds sim parameters whose resulting dynamics are closer the target environment (lower discrepancies), but Table 3 shows the task performance is worse.
Fig. 7 and Fig. 8 further show cases where SysID-Bayes under-performs AdaptSim and there are visible differences between sim parameter distributions found by the two approaches. In the pushing task, SysID- Bayes infers table and patch friction coefficients that are too low, and the trained task policy pushes the bottle with little speed. In the scooping task, interestingly, AdaptSim infers an ellipsoidal shape for the sliced cucumber despite resembling a very thin cylinder, and the task policy achieves 60% success rate. Sys-ID infers cylindrical shape but the task policy fails completely.
Discussions
1.0 2.0
Summary. We present AdaptSim, a framework for efficiently adapting simulation-traine task policies the real world. AdaptSim meta-learns how adapt simulation parameter distributions for better perfor- mance in diverse simulated target environments, and then infers better distributions for training real-world task policies using small amount of real data.
0.99
Poit mose
1.8
516
0.98
0.4 91.
0.5
2.
1.6 3.8
19.0
12
Point
(1..03)
Figure 6: Sim parameters found AdaptSim vs. SysID- Bayes the OOD-1 setting of the double pendulum task. The colors indicate the maximum possible reward at each parameter. SysID-Bayes finds parameters closer to the target in the parameter space (dark red star), but the task performance is worse.
Limitations and Future Work. In some settings AdaptSim does not outperform baselines (e.g., OOD- 4 in the pushing task and scooping up Brussels sprout in hardware, Fig. A8). First, AdaptSim's 's task-driven adaptation training requires the trained task policy being (nearly) optimal on the corresponding simulation parameter distribution - while it can be solved exactly in the double pendulum task, the task policy training in the two manipulation tasks can be noisy. Second, if the target environment is extremely OOD from the simulation domain and the adaptation policy has not been trained with similar trajectories, AdaptSim may not work well. We believe the first issue can be mitigated by allowing more simulation budget for task policy training and better design of task policy re-use. The second issue can be addressed by designing the simulation parameter space Ω to better cover possible real-world behavior.
8

---

Acknowledgments
The authors were partially supported by the Toyota Research Institute (TRI), the NSF CAREER Award [#2044149), and the Office of Naval Research [N00014-23-4 1-2148, N00014-21-1-2803), This article solely reflects the opinions and conclusions of its authors and NSF, ONR, TRI or any other Toyota entity. The authors would like to thank the Dynamics and Simulation team and Dexterous Manipulation team TRI Robotics for technical support and guidance. We also thank Yimin Lin for helpful discussions.
References
[1] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, McGrew, A. Petron, A. Paino, Plappert, G. Powell, R. Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07113. 2019.
[2] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makovicchuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality. arXiv preprint arXiv:2210.13702, 2022.
[3] J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47))eab<5986, 2020.
[4] A. Loquercio, E. Kaufmann,, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza. Deep drone racing: From simulation to reality with domain randomization. IEEE Transactions on Robotics, 36 (1):1-14, 2019.
[5] V. Lim, H. Huang, L. Y. Chen, J. Wang, J. Ichnowski, D. Seita, M. Laskey, and K. Goldberg, Planar robot casting with real2sim2real self- learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2022.
[6] F. Ramos, R. C. Possas, and D. Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. In Proceedings of Robotics: Science and Systems, 2019.
[7] J. Masterjohn, D. Guoy, J. Shepherd, and A. Castro. Velocity Level Approximation of Pressure Field Contact Patches. IEEE Robotics and Automation Letters, 7(4): 1593:11593-11600, 2022.
[8] K--C. Hsu, Z. Ren, D. P. Nguyen, A. Majumdar, and J. Fisac. Sim-to-Labe Laitt--Lab-t--Reee Realto-Rell: Safe reinforcement learning with shielding and generalization guarantees. Artificial Intelligence, 314: 103811, 2023.
[9] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Proceedings of the IEFERSS International Conference on Intelligent Robots and Systems (IROS), 2017.
[10] F. Sadeghi and S. Levine. Cad2rl: Real single-image flight without single real image. In Proceedings of Robotics: Science and Systems, 2017.
[1]] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018
[12] G. B. Margolis and P. Agrawal. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior. In Proceedings of the Conference Robot Learning (CoRL), 2022.
[13] F. Muratore, F. Ramos, G. Turk, W. Yu, M. Gienger, and Peters. Robot learning from randomized simulations: A review, Frontiers in Robotics and AI, 2022.
[14] M. Gautier and W. Khalil. On the identification of the inertial parameters of robots. In Proceedings of the IEEE Conference Decision and Control (CDC), 1988.

---

[15] P. K. Khosla and T. Kanade. Parameter identification of robot dynamics. In Proceedings of the IEEE Conference on Decision and Control (CDC), 1985.
[16] E. Heiden, E. Denniston, D. Millard, F. Ramos, and G. Sukhatme. Probabilistic Inference of Simulation Parameters via Parallel Differentiable Simulation. In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), 2022.
[17] R. Antonova, J. Yang, P. Sundaresan, D. Fox, F. Ramos, and J. Bohg. A bayesian treatment of real-to- sim for deformable object manipulation. IEEE Robotics and Automation Letters, 733)5581--5822 2022.
[18] C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Iterative residual policy: for goal- conditioned dynamic manipulation of deformable objects. In Proceedings of Robotics: Science and Systems, 2022.
[19] F. Muratore, C. Eilers, M. Gienger, and J. Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters, 6(2):911 1-991, 2021.
[20] Q. Vuong. S. Vikram, H. Su, S. Gao, and H. I. Christensen. How pick the domain randomization pa- rameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint 177744 2019.
[21] W. Yu, C. K. Liu, and G. Turk. Policy transfer with strategy optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[22] N. Ruiz, S. Schulter, and M. Chandraker. Learning to simulate. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[23] A. Bensoussan. Stochastic control of partially observable systems. Cambridge University Press, 1992.
[24] J. Swevers, C. Ganseman, D. B. Tukel, J. De Schutter, and H. Van Brussel. Optimal robot excitation and identification. IEEE Transactions on Robotics and Automation, 13(5)7330-740, 1997.
[25] A. Tavakoli, F. Pardo, and P. Kormushev. Action Branching Architectures for Deep Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[26] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference artificial intelligence, 2016.
[27] R. Tedrake. Underactuated Robotics. 2023. URL htttps://underacuuateee csail mit. edu.
[28] Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL https://draak: mit. edu.
[29] R. Antonova, F. Ramos, R. Possas, and D. Fox. BayesSimlG: Scalable Parameter Inference for Adaptive Domain Randomization with Isaac Gym. arXiv preprint arXiv:2107.04527, 2021.
[30] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratifff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In Proceedings of the IEEE International Conference on Robotics and Aulomation (ICRA), 2019.
[31] A. Allevato, E. S. Short, M. Pryor, and A. Thomaz. Tunenet: One-shot residual tuning for system identification and sim-to-real robot task transfer. In Proceedings of the Conference on Robot Learning (CoRL), 2020.
[32] Ajay, M. Bauza, J. Wu, N. Fazeli, J. B. Tenenbaum, A. Rodriguez, and Kaelbling. Combining physical simulators and object- based networks for control. In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), 2019.
10

---

[33] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4): 2020.
[34] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid motor adaptation for legged robots. In Proceedings of Robotics: Science and Systems, 2021.
[35] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification, In Proceedings of Robotics: Science and Systems, 2017.
[36] B. Evans, A. Thankaraj, and L. Pinto. Context is everything: Implicit identification for dynamics adaptation. In Proceedings of the IEEE Interational Conference on Robotics and Automation (ICRA), 2022.
[37] J. Liang, S. Saxena, and O. Kroemer. Learning active task-oriented exploration policies for bridging the sim-to-real gap. In Proceedings of Robotics: Science and Systems, 2020.
[38] W. Jin and M. Posa. Task-Driven Hybrid Model Reduction for Dexterous Manipulation. arXiv preprint arX6v:2211.16657, 2022.
[39] A. Z. Ren and A. Majumdar. Distributionally robust policy learning via adversarial environment generation. IEEE Robotics and Automation Letters, 7(2):1379-1138)) 2022.
[40] H. S. Gomes, B. Léger, and C. Gagné. Meta learning black-box population-based optimizers. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[41] M. Wistuba, N. Schilling, and L. Sechmidt-Thiemss Scalable gaussian proces--basd transfer surrogates for hyperparameter optimization. Machine Leurning, 107(():43-78, 2018.
[42] Y. Chen, M. W. Hoffman, G. Colmenarejo, M. Denil, T. P. Lillicrap, M. Botvinick, and N. Freitas. Learning to learn without gradient descent by gradient descent. In Proceedings of the International Conference Machine Learning (ICMI), 2017.
[43] M. Volpp, L. P. Fröhlich, K. Fischer, A. Doem,, S. Falkner, Hutter, and C. Daniel. Meta-learning acquisition functions for transfer learning bayesian optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[44] A. Z. Ren, B. Govil, T.-Y. Yang, K. Narasimhan, and A. Majumdar. Leveraging Language for Accelerated Learning of Tool Manipulation. In Proceedings of the Conference on Robot Learning (CoRI.), 2022.
[45] I.. Johannsmeier, M. Gerchow, and S. Haddadin. A framework for robot manipulation: Skill formalism, meta learning and adaptive control. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2019.
[46] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2018.
[47] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement leamnin. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[48] S. Gu, T. Lillicrap, I. Sutskever, and S. Levine. Continuous deep q-learning with model-based acceleration. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
11

---

Appendix
A1 Extended Related Work
Sys-ID domain adaptation. Inspired by classical work in Sys-ID [14, 15], there has been a popular line of work identifying simulation parameters that match the robot and environment dynamics in the real environment before task policy training. BayesSim [6] and follow-up work [16, 17] applies Bayesian inference to iteratively search for posterior distribution of the simulation parameters based on simulation and real-world trajectories. The inference problem has also been formulated using RL to minimize trajectory discrepancies [30]. A different approach [31, 32, 33] learns a residual model of dynamics (often parameterized with a neural network) to match simulation or an ideal physics model with reality. However, all these methods consider relatively well-modeled environment parameterizations such as object mass or friction coefficient during planar contact; Sys-ID approaches have been shown to fail in cases where the simulation does not closely approximate the real world [13, 18]. There is also work that avoids inferring the full dynamics but adapts with low-dimensional latent representation online [34, 35, 36), but the representation is still trained with regression to match dynamics or simulation parameters. Importantly, the Sys-ID approaches highlighted above are all task-agnostic; this can lead to poor performance when trained task policies are sensitive to mismatches dynamics between simulation and reality. Chi et al. [18] address the issue by using simulation to predict changes trajectories from changes in actions as an implicit policy, but it requires the environment to be resettable, while AdaptSim works with randomly initialized object states.
Task-driven domain adaptation. AdaptSim better fits within a different line of work that aims to find simulation parameters that maximize the task reward in target environments. Muratore et al. [19] apply Bayesian Optimization (BO) to optimize parameters such as pendulum pole mass and joint damping coefficient real pendulum swing-up up task. Other work focus on adapting to simulated domains only [20, 21, 22]. One major drawback of these methods is that they require large number of rollouts in target environments (e.g.. 700 in [19]), which is very time-consuming for many tasks requiring human reset. AdaptSim meta-learns adaptation strategies in simulation and requires only few real rollouts for inference (e.g., 20 in our pushing experiments). Liang et al. [37] apply the same task-driven objective to learn an exploration policy in manipulation tasks, but the task policy is synthesized using estimated simulation parameters via Sys-ID. Jin al. [38] applies task-drived reduced-order model for dexterous manipulation tasks, but again the model is identified with Sys-ID and no vision-basse control is involved. Ren et al. [39] search for adversarial environments (e.g., objects) given the current task performance to robustify the policy, but unlike AdaptSim, the adversarial metric is measured in simulated domain only without real data.
Learn to search/optimize. Our work involves learning optimization strategies through meta-learning across a distribution of relevant problems, allowing for customization to the specific setting and increased sample efficiency [40, 41]. Chen al. [42] meta-learns an RNN optimizer for black-box optimization. Volpp et al. [43] meta-leams the acquisition function in BO with RL; it able learn new exploration strategies for black-box optimization and tuning controller gains in sim-to-real transfer. Meta RL trains the task policy directly optimize performance in new environments [44, 45, 46] AdaptSim applies meta
RL to optimize simulation parameters instead.
A2
Additional details on approach
A2.1 Sparse adaptation reward
In practice, we are only concerned with the reward if it reaches some minimum threshold - bad task policy is not useful. Thus we use a sparse-reward version of Eq. (2),
E E (x'1
(AI)
Eor
==0
where 1() is the indicator function and R is the sparse-reward reward threshold. Using sparse reward also discourages the adaptation policy from being myopic and getting trapped at sub-optimal solution,
12

---

especially since we use relatively small I (e.g, 5-10) in order to minimize the amount of real data, and use a small discount factor (=0.).
A2.2 Task policy reuse across parameter distributions
Algorithm requires training the task policy for each &, which can be expensive with the two manipulation tasks. Our intuition is that share the task policy between parameter distributions of close distance, with the following heuristies:
Record the total budget (i.e., number of trajectories), and j. the number of simulation parameter distributions that task policy has been trained with.
Define distance between two parameter distribution D(-,)) such as L2 distance between the mean. If is within a threshold D from a previously seen distribution, re-use the task policy. If the policy is already trained with Mmax budget total, do not train again; otherwise train with max (Mmin; M) budget, where <<1 1 and M is the budget for training the policy for the first time.
If the nearby parameter distribution re-uses a task policy, do not re-use the same policy again. This prevents the same task policy being used for too many E.
Remark 1 re-using task policies between parameter distributions makes the reward R depend on the aduptation history, as nie depends on previous & that are used for training. We choose not to model this history dependency in f, as the reward should be largely dominated by the current E
A3 Additional details of adaptation policies
Hyperparameters. Table Al shows the hyperparameters used for the adaptation policy training in Phase 1, including those defining the heuristics for re-using task policies among simulation parameter distributions. We generally use smaller adaptation step 5 for smaller dimensional Ω
Task
Parameter
Pendulum Pushing Scooping
Total adaptation steps, K
le4
le4
le4
Adaptation horizon, I
8
Adaptation step size, 8
0.10
0.15
0.15
Adaptation discount factor, Sprase reward threshold,
0.9
0.9
0,9
0.95
0.8
0.5
Task policy reuse threshold, D
0.16
0.16
Task policy max budget, Mms
3e4
4e3
Task policy budget discount,
0.9
0.9
Task policy init budget, M
le4
1.2e3
Table Al: Hyperparameters used in adaptation policy training for the three tasks.
Trajectory observations. We detail the trajectory observation (as input to the adaptation policy) used in the three tasks.
Pendulum task: each trial is 2.5 seconds long, and use 12 evenly spaced points along the trajectories of the two joints, and thus each trajectory is 24 dimensional. For AdaptSim- again 12 points are used but sampled from the last 0.5 second only. One trajectory is used at each adaptation iteration - the trajectory input to the adaptation policy is 24 dimensional.
BesID-Bayes-Suae and
Pushing task: each trial is 1.3 seconds long, and we use evenly spaced points along the X-Y trajectory of the bottle, and thus each trajectory also 12 dimensional. For AdaptSim-State, SysiD-Bayes-Stain and SysID-BB BEDD--Byes-Pooo Point, only the final X-Y position of the bottle is used. Two trajectories are used at each adaptation iteration the trajectory input the adaptation policy is 24 dimensional.
13

---

Scooping task: each trial is 1 second long, and we use X-Y position of the food piece at the time step 0,1.2.0.3.044005,00.,..o 1 .OJs (more sampling around the initial contact between the spatula and the piece), and thus each trajectory is 16 dimensional. Two trajectories are used at each adaptation iteration - the trajectory input to the adaptation policy 32 dimensional.
In real experiments, we track the bottle position the pushing task using 3D point cloud information from Azure Kinect RGB-D camera, which we find accurate. In the scooping task, the food pieces are too small and thin to be reliably tracked with point cloud, and thus we resort extracting the contours from the RGB image and then finding the corresponding depth values at the same pixels in the depth image. During fast contact there can be motion blur around the food piece, and thus we add Gaussian noise with 0.2cm mean for X position and zero mean for Y position, and 0.2cm covariance for both, to the points in the ground-truthe trajectories in simulation. We use positive mean in X since the motion blur tends to occur in the forward direction.
A4 Additional details of the task setup and task policies
Trajectory observation First, we remove the action sequence from the task-policy trajectory and keep the state sequence only. Since the dynamics real environments can be OOD, in order to achieve similar high-reward states as simulated environments, the robot would need to use some actions not seen during training (or not seen for the particular state), hindering the adaptation policy to generalize if action sequence were included in the task policy trajectory. We assume that the task-relevant state sequence is covered by T if the task policy performs reasonably well in the real environment. This choice is also present in the state-only inverse RL literature [47] that addresses train-test dynamics mismatch. See Fig. A4 and related discussions in Sec. 6.3.
A4.1 Dynamic pushing of bottle
Trajectory parameterization. Here we detail the trajectory of the end-effector pusher designed for the task (Fig. Al). The trajectory is parameterized with two parameters: (I) planar pushing angle, which is the yaw orientation of the pusher relative to the forward direction that controls the direction of the bottle being pushed, and (2) forward speed (of the end-effector), in the direction specified by the pushing angle. The pushing angle varies between -0.3rad and 0.3rad, and the forward speed varies between 0.4m/s and 0.8m/s. We find 0.8m/s roughly the upper speed limit of the Franka Panda arm used. The pusher also pitches upwards during the motion and the speed is fixed to 0.8rad/s. We design such trajectories to maximize the pushing distance at the hardware limit.
Initial and goal states. The bottle placed at the fixed location (x=0.56m.y-0, relative to the arm base) on the table before the trial starts. The goal location is sampled from region where the X location is between 0.7 and 1.0m and Y location is at most 10 degrees off from the centerline (Fig. A1 top-right). The patch, 10cm by 10cm square, is placed at x 0.75m with its center (lateral position is varied as one of the simulation parameter).
Task policy parameterization. The task policy is parameterized using Normalized Advantage Function (NAF) [48] that allows efficient Q Learning with continuous action output by restricting the Q value as quadratic function of the action, and thus the action that maximizes the Q value can be found exactly without sampling. In this task, it maps the desired 2D goal location of the bottle to the two action parameters, planar pushing angle and forward speed. The policy is open-loop the actions are determined before the trial starts and there is no feedback using camera observations.
Hardware setup. A 3D-printed, plate-like pusher is mounted the end-effector instead of the paralle-jaw gripper in both simulation and reality. We also wrap elastic rubber bands around the bottom of the pusher and contact regions of the bottle induce more elastic collision, which we find increases the sliding distance of the bottle.
14

---

Side view Initial
Side view Final
Top- down view
Forward speed
Planar pushing angle
Pitch speed fixed
Goal zone
Figure A1: Visualization the pushing trajectory and goal locations the Drake simulator. There are two action parameters: (I) forward speed (of the end-efflctor) and (2) planar pushing angle (i.e., yaw orientation of the end- effector). The patch is not visualized.
Side
Top
Pitch rate
Intiall
position
distance
-0.4
angle
-0.
00
0.4
0.6
Figure A2: Visualization of sconping trajectory and initial positions of food piece the Drake simulator. There are three action parameters: (1) initial distance (between the spatula and food piece). (2) initial pitch angle (of the spatula from the table), and (3) pitch rate (of the end-effector at time step t= 0.25)..
A4.2 Dynamic scooping of food pieces
Trajectory parameterization. Here we detail the trajectory of the end-effector with the spatula designed for the task (Fig. A2). The end-effector velocity trajectory is generated using cubic spline with values clamped at five timesteps. The trajectory only varies the and pitch direction (in the world frame), while remaining zero in the other directions. The only value defining the trajectory that the task policy learns is the pitch rate, which is the pitch speed at the time t-0.25s and varies between -0.2rad/s and 0. 0.2rrd/s. A positive pitch rate means the spatula lifting off the table late, while negative means lifting off early (see the effects in Fig. 8). The other two values that the task policy outputs are the initial pitch angle of the spatula from the table (varying from 2 to 10 degrees), and the initial distance between the spatula and the food piece (varying between 0.5cm to 2cm). Generally a higher initial pitch angle can help scoop under food pieces with flat bottom, and a smaller angle helps scoop under ellipsoidal shapes. We design such trajectories after extensive testing with food pieces of diverse geometric shapes and physical properties both simulation and reality.
Initial states, The food piece is randomly placed in box area of 8x6em in front of the spatula; the initial distance is relative to the initial food piece location.
Task policy parameterization. The task policy is parameterized using a NAF again. In this task, maps the initial 2D position of the food piece to the three action parameters: pitch rate, initial pitch angle, and initial distance.
Hardware setup. We use the commercially available OXO Nylon Square Turner" as the spatula used for scooping. has a relatively thin edge (about 1.2mm) that helps scoop under thin pieces. A box-like, 3D 3D-printed adapter with high-friction tape is mounted on the handle to help the parallel-jaw gripper grasp the spatula firmly. The exact 3D model of the spatula with the adapter is designed and used in the Drake simulator; the deformation effect bends against the table is not modeled in simulation.
link: https: //www/ amazon. com -x0 1tt.00111000000000drr-----------
15

---

A5
Additional details of experiments
A5.1
Simulated adaptation
Table A2 shows the simulation parameters used different simulated target environments for the three tasks (results shown Table 3).
Setting
Task
Parameter WD OOD-1 OOD-2
OOD-3 OOD-4
Range
1.8
0.5
1.2
[1.2]
1.2
0.3
1.8
1.8
2.6
[1,2]
Pendulum
1,5
1.5
1.5
10.0
1.0
[1,2]
bz
1.5
1.5
1.5
10.0
2.0
[1,2]
0.1
0.25
0.05
0.15
0.30
[0.05.0.2]
le5
5e4
le5
5e6
le5
[164,1e6
Pushing
0.6
0.1
0.9
0.1
0.15
[0.2,0.8]
Mp
0.05
-0.1
0.05
-0.15
0.1
[-0.1,0.1]
0.30
0.45
0.20
0.30
0.40
[.25,,04
5e4
le4
5e4
1c6
le5
[1e4,5e5]
Scooping
1
0
0
2
{0,1)
2.0
2.2
1.9
[1.5,2.5]
Table A2: Simulation parameters used in different simulated target environments three tasks. OOD parameters (outside the range used in adaptation policy training) are bolded. For in the scooping task, 0 stands ellipsoid, for cylinder, and 2 for box.
A5.2 Real adaptation
In Fig. A7 and Fig. A9 we demonstrate additional visualizations the pushing and scooping results with AdaptSim.
A5,3 Additional studies
Choice of the simulation parameter space. To answer Q3, we perform sensitivity analysis by fixing the target environment (OOD-1 in the double pendulum task) and varying the simulation parameter space. In 00D-1, the OOD parameter is m2 =0.3 while the range in Ω is [1,2]. Fig. A3 shows the results of reward achieved after adaptation for AdaptSim and the two Sys-ID baselines, as the range shifts further away from m2 = 0.3 to [1.,, [1.2,2.]] and [1.3,2.3] Sys-ID performance degrades rapidly, while AdaptSim is more robust.
Normalized reward
Adapetim
TydID-Sayes-iial
SydDD Poin--Po-----raa
0.8
0.2
[.00.
[11,
[1.2,
[1.3,
Figure A3:
Adaptation results for AdaptSim and Sys-ID baselines in OOD-1 setting of the double pendulum task, with different ma ranges in S while m/ -0.3 in the target environment.
Pitfalls of Sys-ID approaches. Fig. A4 demonstrates the dynamics mismatch between simulation and reality, which illustrates the pitfall of SysID approaches. We plot a set of bottle trajectories from randomly sampled simulation parameters from 12 with fixed robot action. We also plot the trajectories of Heavy bottle being pushed with the same action in reality. There are segments of real trajectories that are well matched by the simulated ones, and slight mismatch can lead to diverging final states (and hence different task rewards).
16

---

1.1
0.10
0.08
80s (m)
0.9
(m)
0.08
0005
0.8
0.04
0.7
0.02
Sim
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
Time (s)
Figure A4: Comparison of trajectories from the simulation domain (green, simulated with randomly sampled simulation parameter settings) and from Heavy bottle reality (red), with same robot action applied. The real dynamics can be OOD from simulation (black boxes) while the final position the bottle can be WD.
Trade-off between real data budget and task performance convergance. In Sec. 4.2 we introduce N, the number of initial simulation parameter distributions that are sampled at the beginning of Phase and then adapt independently. There is trade-off between the real data budget (linear N) and convergence of task performance. Adapting more simulation parameter distributions simultaneously can potentially help the task performance converge faster but also require more real data. Fig. A5 shows the effect with the Light bottle in the pushing task. We vary N from I to 4 - each simulation parameter distribution takes trajectories at each iteration. -= 1 shows slow and also worse asymptotic convergence, which shows that the parameter distribution can be trapped in a low-reward regime. = 2 performs the best with fastest convergence in terms of number of real trajectories used. Using higher N shows slower convergence. Note that the convergence also depends on the dimension of the simulation parameter space Q N > 2 is needed for the best convergence rate once the dimension increases from 4 used in the pushing task.
expect
1.0
Normalized reward
N=1
0.9
N
N=4
0.8
0.7
0.6
8
12 14 16
18 20 22 24
Number of real trajectories
Figure A5: Task performance convergence with respect the number of real trajectories used with varying the number simulation parameter distributions adapting simultaneously in Phase with the Light bottle in the pushing task.
Sensitivity analysis on adaptation step size. Adaption step size 8 can affect the task performance convergence too - 8 being too low can cause slow convergence, while 6 being too high can prevent convergence since the simulation parameter distribution can "overshoot" the optimal one by a large margin. Fig. A6 shows the effect of adaptation step size ranging from 0.05 0.20 in OOD-1 setting of the double pendulum task. 8 = 0.10 performs the best while 5 0.05 shows slower convergence. 8 = 0.15 also achieves similar asymptotic performance but the reward is less unstable during adaptation, while with 5 5- 0.20 the reward does not converge at all.
Comparison of simulation runtime. Compared to Sys-ID baselines, AdaptSim requires significantly longer simulation runtime for training the adaptation policy Phase For example: SyslD- uses roughly hours of simulation walltime perform 10 iterations of adaptation in the scooping task while AdaptSim would take 36 hours for Phase and 30 minutes for Phase 2 (i.e., minutes per iteration), using the same computation setup. However, we re-use the same adaptation policy for different food pieces in the scooping task, which amortizes the simulation cost.
17

---

1.0
Normalized reward
0.8
6=0.05
0.6
6=0.10
6=0.15
0.5
6-0.20
0.4
2
4
6
7
9
10
Iterations
Figure A6: Normalized reward at each adaptation iteration using different adaptation step size in OOD-1 setting of the pendulum task.
Iteration
Iteration
Iteration
0.20
Table fiction confficient,
Hydroelastic modulus
0.10
0.01
Ratch friction coefficient
Patch lestral position Im
0.75
0.1
AdapiSim
0.50
0.0
Iterations
Figure A7: Adaptation results of the pushing task with two different target locations (ycllow cross, top and bottom rows) over iterations. The right figure shows the inferred simulation parameter distribution (mean only).
Figure A8: AdaptSim fails to synthesize task policy for scooping up Brussels sprout. We consider such environment extremely OOD from the simulation domain.
18

---

Iteration
Iteration 2
Iteration 3
a
Iteration
Beel
Iteration
Iteration
Iteration 2
Iteration 3
Iteration
ays
Iteration
Iteration
Iteration
Iteration
Iteration
Iteration
Figure A9: Adaptation results of scooping up (top) chocolate raisins, (middle) mushroom slice, and (bottom) Oreo cookie with AdaptSim.
19


Reviews:
[{"review_id": "xfJlprM7EiB", "reviewer": "Reviewer_XPgu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new method for sim2real transfer. It first learns an adaptation policy in simulation, which outputs the delta change in simulation parameters to maximize the policy performance in the target environment, and then use the learned adaptation policy in the real world. This idea differs from traditional sim2real methods where the goal is to match the simulation parameter to the real-world ones. The method is evaluated on simulation to simulation adaptation tasks and also 3 simulation to real-world adaptation tasks. The method is shown to outperform several previous baselines including system ID and domain randomization methods.", "summary_of_recommendation": "See above. I am happy to raise score if my concerns are addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Lack of comparison / discussion to similar prior works, such as SimOpt [1] and TuneNet [2]. The idea seems very similar to these 2 prior works. The authors should discuss the relationships between these two, and include these as baselines in the experiments. \n\nDiscussion on why the adaptation policy itself is not having a sim2real issue and how are the real-world object locations obtained.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n- The problem of sim2real transfer is of important to the community and is thus well-motivated.\n- The paper is clearly written and easy to follow.\n- The real-world experiments are well designed such that sim2real gaps do exist, and some good analyses are performed to better understand why the methods work. The high-velocity scooping task is non-trivial to solve and it's good to see the method works on it.\n\nWeakness:\n- Lack of comparison / discussion to similar prior works, such as SimOpt [1] and TuneNet [2].\n- It requires access of ground-truth object locations (e.g.,  evenly spaced points along 2D trajectories of the food pieces) in the real world as observations, which can be hard to track for small, arbitrary shape food pieces. \n- The adaptation policy is trained fully in simulation. Therefore, itself faces a sim2real problem. The authors are encouraged to add more discussion on why this adaptation policy would transfer better to the real-world compared to the task policy.\n\n[1] Chebotar et al, Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience\n\n[2] Allevato et al, TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "w_h56uo3XJC", "reviewer": "Reviewer_zQUg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper describes a Sim2Real approach based on adapting simulation parameters to match real-world target tasks. The simulation parameters are adapted in order to facilitate training of a policy, which aims to perform well on the real world task, rather than aiming to exactly match the transition dynamics between the simulation and real-world environments. This allows the proposed approach to be used in scenarios where the dynamics cannot be accurately modeled, such as contact-rich manipulation tasks. ", "summary_of_recommendation": "While some questions need to be answered by the authors in order for me to be fully convinced that this paper is complete, the approach described in the paper seems sound and the experiments are comprehensive, displaying a successful application of the approach. Therefore, I believe this paper constitutes a meaningful contribution to the robotic learning community, and should probably be accepted to CoRL. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "An additional question (beyond the ones in the body of the review):\n\nWhat are some domains where the “irreducible” gap also prevents optimizing for the real-world domain? This is mentioned in passing in the limitations section (the Brussels sprout task is “too OOD” from the simulation domain), but would be good to see a more comprehensive analysis of the scope of this limitation. How should we choose the distribution of simulation parameters? Are there real-world tasks this approach would never be able to solve?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and clearly explains its methods and experiments. In particular, the related work section (and especially the extended version of it in the appendix) is comprehensive and covers many similar approaches in the sim2real and SysID area. The authors do a good job of differentiating between the variety of existing methods, and explaining why their method is different. \n\nThe experiments conducted by the authors are thorough and demonstrate their approach well. I appreciate the complexity of the manipulation tasks chosen. While they do not cover a wide range of manipulation tasks, both of the real-robot environments showcase the idea of “irreducible” gaps between sim and real. In a more general sense, it is very cool to see the scooping task solved at all - it seems like this would be a challenging task even for a human operator. \n\nWhile the approach described in the paper seems useful and practical for some robotic domains, several questions and concerns arise when reading through the approach and experiment sections. \n\n1. The task policy is trained at every iteration; is this not an extremely expensive operation? It seems like this could only work for tasks where the policy can be very simply parameterized with a few parameters. What happens if the task policy needs to be more complex, and doesn’t converge for the specific set of sim parameters?\n    \n    A similar concern is regarding the training of the adaptation policy in phase 1; does it also require the trajectory representations to be relatively low dimensional? From Sec. A3, it seems like all trajectory observations include a relatively small number of points. Would this approach still be feasible for tasks which require a more expressive representation of trajectories? \n    \n\n1. The LearnInTarget baseline presented in the paper seems slightly too simplistic. Wouldn’t it be a more correct comparison to train the task policy in the target domain to convergence, or at least in the same manner as the training of the task policy in the inner step of phase 1? I expected the LearnInTarget baseline to be an “upper bound” policy, demonstrating what the optimal reward would be in the environment. \n\n1. While BayesSim and its follow-up work were originally trained for well-modeled environment parameters, wouldn’t they still be applicable to the setting described in this paper? A comparison of the proposed approach to BayesSim would be a welcome addition to the experiments in the paper. In general, the paper would benefit from a comparison to one of the adaptive (domain-randomization based?) SysID approaches described in the related work. Baselines include a uniform DR approach and a SysID approach, but not combinations of the two.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "NeXCJvASQn", "reviewer": "Reviewer_bMZi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents AdaptSim, a method for Sim-to-Real transfer. Unlike the conventional approaches based on system identification (SysID), AdaptSim learns environment parameters that induce maximize task rewards in the target environment. AdaptSim involves a two-step learning procedure. First, it learns a meta adaptation policy via reinforcement learning that iteratively updates the environment parameters based on trajectories observed in the environment. This meta adaption policy is trained on a wide range of environments in simulation. Next the method is deployed to the real world environment, where environment parameters are iteratively updated using the meta adaption policy and new policies are evaluated in the real world environment. On a number of simulated and real world environments, the authors demonstrate that AdaptSim outperforms existing Sim-to-Real approaches, particularly SysID approaches in settings where the simulator cannot fully model the target environment.", "summary_of_recommendation": "Overall this is a well executed paper, with a well-motivated introduction, detailed presentation of the method, and thorough experiments. That said, I have two major questions regarding the underspecification problem and the task-agnostic nature of the meta adaption policy f. Looking forward to hearing clarification from the authors. \n\nUpdate after rebuttal: I read the other reviewers' comments and the author responses to those reviews. Seems to most reviewers had an initially high sentiment and that that sentiment remains the same after the author responses. I appreciate that the authors took steps to clarify the questions the reviewers had. That said, they made very limited changes to the manuscript and some of the limitations that the reviewers raised (having access to ground truth information, underspecification problem related to maximizing rewards) still remain. Therefore right now I would like to keep my recommendation as is, weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See the last two points in the previous section", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is overall a well-executed paper. The strengths are as follows:\n* Well-written introduction, with clear motivations and explanations. Paper makes the problem setting clear from the get-go (Sim-To-Real transfer with irreducible gaps)\n* The method section provides a complete description of the problem statement and the proposed method. Pseudocode help to enhance readability. \n* Experiments are thorough and provide convincing evidence of the benefits of the proposed approach. There are adequate quantitative and qualitative results, with a strong emphasis on real robot results.\n\nThere are some key questions I have (these should not necessarily be viewed as weaknesses, but potential limitations worth addressing):\n* The goal here is to find environment parameters that maximize the task rewards, however is this not a underspecified problem in general? Especially if the reward function is very simple, then compared to the SysID approaches this approach may find parameters that do not necessarily reflect the true parameters of the environment (in fact this seems to be acknowledged in figure 6). This may be ok if we are transferring single-task policies for limited evaluation settings. However in more general cases we may want general-purpose policies that are robust to perturbations, and work under large initial state distributions. In these cases, it would be important that the estimated parameters are not merely inferred through correlation with task reward, but rather reflective of the actual dynamics of the environment (causal).\n* The meta adaption policy f is trained to maximize rewards under the exception of all environments (see the outer expectation in equation 2). This implies that f is a task-agnostic policy. However, we want to use f to iteratively update parameters for a specific task. Thus I am confused why f should necessarily find the optimal task parameters when it is not aware of the target task reward function. Seeking clarification on this. Please provide additional intuition on why f enables us to find the optimal environment parameters. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "HpanYUsb2lM", "reviewer": "Reviewer_FEC8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method for adapting a simulator to enable good task performance in the real world in the context of learning policies via reinforcement learning. The key idea is to find simulation parameters such that the optimal policy in an environment corresponding to those simulation parameters will perform the best in the real world. This is done by formulating a meta-learning problem where a number of target environments are sampled in simulation, and a network is learned to adapt initial simulation parameters so a trained policy can achieve high performance in these target environments. This network is then used to adapt initial simulation parameters to achieve high performance in an arbitrary real world target environment at test time.", "summary_of_recommendation": "I am generally positive on this paper and am leaning towards acceptance. The presented method is clearly novel and it is well motivated why it should outperform prior works on sim to real adaptation. I think the main weakness of the paper currently is the lack of clear motivation about the precise conditions under which one would expect AdaptSim to outperform SysID methods, which I believe could be addressed by adding an experiment with a toy example as discussed above.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "During meta-training, the target environments are sampled fully in simulation and a network is learned to adapt simulation parameters to achieve good performance in these target environments. Since the proposed approach is best suited for situations in which the true target environment (real world) is OOD from the simulation environments, learning to adapt to the target environments sampled at meta-train time seems like it would not necessarily be helpful for adapting towards the true target environment. Essentially, there is no irreducible error at meta-train time. Would adapt-sim benefit from constructing target environments at meta-train time so that some irreducible error is guaranteed (ie. by only adapting some portion of the sim parameters instead of all of them?). \n\nWould it be possible to add an experiment studying the sensitivity of AdaptSim to the adaptation horizon I? I suspect very large adaptation horizons could lead to some overfitting.\n\nHow exactly are simulation parameters sampled for the UDR baseline? For a fair comparison, since parameters are updated over discrete intervals for AdaptSim, a comparison with UDR over parameters sampled over the same grid would be helpful.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting idea and is written very clearly. The idea is to find the ideal distribution of simulation parameters to train a policy that will perform well in some OOD target environment, regardless of whether those simulation parameters actually provide an accurate model of the target environment. This is in contrast to domain randomization techniques, which simply assume this distribution of simulation parameters is uniform, and sys ID techniques, which assume this distribution should just be a point estimate where the simulation parameters are set such that the dynamics most closely match reality. \n\nStrengths: (1) the method seems to be a novel approach for sim to real transfer that achieves strong results compared to relevant prior work; (2) the idea of the method is clearly motivated, and it is clear why in principle it should be able to outperform previous methods\n\nWeaknesses: (1) there is not sufficient discussion of how the task policy is trained for a given set of simulation parameters in the main text, this seems like a sufficiently important part of the method that it should not be in the appendix; (2) the paper would benefit from a concrete and simple motivating example illustrating some specific situations in which the simulation parameters that lead to good task performance may be different from simulation parameters that most closely match the real world. While there are empirical results suggesting this in section 6.3, it would be very helpful to study a simple case where we can exactly compute the simulation parameters that most closely match reality (of course for a simulation in which reality is OOD) and show that the best parameters for training a policy in simulation to achieve high reward in the real environment are actually not those parameters.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a method to learn adaptation policies in simulation to maximize their performance in a target environment which can be deployed to the real robot. As opposed to conventional sim2real methods, this approach is focused on the policy performance in the target environment and not necessarily attempts to capture the real dynamics of the system in simulation. 

The reviewers agree that the paper makes a good contribution, is well written and motivated, and presents solid analyzes. The rebuttal helped to clarify specific issues, such as the need for comparisons to SimOpt. Authors are encouraged to include as many suggestions as possible in the final version. In particular, elaborate on the need to access ground truth information and the underspecification problem related to maximizing the rewards.
Decision: Accept (Poster)