Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a new method for sim2real transfer. It first learns an adaptation policy in simulation, which outputs the delta change in simulation parameters to maximize the policy performance in the target environment, and then use the learned adaptation policy in the real world. This idea differs from traditional sim2real methods where the goal is to match the simulation parameter to the real-world ones. The method is evaluated on simulation to simulation adaptation tasks and also 3 simulation to real-world adaptation tasks. The method is shown to outperform several previous baselines including system ID and domain randomization methods.
review: Strength:
- The problem of sim2real transfer is of important to the community and is thus well-motivated.
- The paper is clearly written and easy to follow.
- The real-world experiments are well designed such that sim2real gaps do exist, and some good analyses are performed to better understand why the methods work. The high-velocity scooping task is non-trivial to solve and it's good to see the method works on it.

Weakness:
- Lack of comparison / discussion to similar prior works, such as SimOpt [1] and TuneNet [2].
- It requires access of ground-truth object locations (e.g.,  evenly spaced points along 2D trajectories of the food pieces) in the real world as observations, which can be hard to track for small, arbitrary shape food pieces. 
- The adaptation policy is trained fully in simulation. Therefore, itself faces a sim2real problem. The authors are encouraged to add more discussion on why this adaptation policy would transfer better to the real-world compared to the task policy.

[1] Chebotar et al, Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

[2] Allevato et al, TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: See above. I am happy to raise score if my concerns are addressed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents AdaptSim, a method for Sim-to-Real transfer. Unlike the conventional approaches based on system identification (SysID), AdaptSim learns environment parameters that induce maximize task rewards in the target environment. AdaptSim involves a two-step learning procedure. First, it learns a meta adaptation policy via reinforcement learning that iteratively updates the environment parameters based on trajectories observed in the environment. This meta adaption policy is trained on a wide range of environments in simulation. Next the method is deployed to the real world environment, where environment parameters are iteratively updated using the meta adaption policy and new policies are evaluated in the real world environment. On a number of simulated and real world environments, the authors demonstrate that AdaptSim outperforms existing Sim-to-Real approaches, particularly SysID approaches in settings where the simulator cannot fully model the target environment.
review: This is overall a well-executed paper. The strengths are as follows:
* Well-written introduction, with clear motivations and explanations. Paper makes the problem setting clear from the get-go (Sim-To-Real transfer with irreducible gaps)
* The method section provides a complete description of the problem statement and the proposed method. Pseudocode help to enhance readability. 
* Experiments are thorough and provide convincing evidence of the benefits of the proposed approach. There are adequate quantitative and qualitative results, with a strong emphasis on real robot results.

There are some key questions I have (these should not necessarily be viewed as weaknesses, but potential limitations worth addressing):
* The goal here is to find environment parameters that maximize the task rewards, however is this not a underspecified problem in general? Especially if the reward function is very simple, then compared to the SysID approaches this approach may find parameters that do not necessarily reflect the true parameters of the environment (in fact this seems to be acknowledged in figure 6). This may be ok if we are transferring single-task policies for limited evaluation settings. However in more general cases we may want general-purpose policies that are robust to perturbations, and work under large initial state distributions. In these cases, it would be important that the estimated parameters are not merely inferred through correlation with task reward, but rather reflective of the actual dynamics of the environment (causal).
* The meta adaption policy f is trained to maximize rewards under the exception of all environments (see the outer expectation in equation 2). This implies that f is a task-agnostic policy. However, we want to use f to iteratively update parameters for a specific task. Thus I am confused why f should necessarily find the optimal task parameters when it is not aware of the target task reward function. Seeking clarification on this. Please provide additional intuition on why f enables us to find the optimal environment parameters. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall this is a well executed paper, with a well-motivated introduction, detailed presentation of the method, and thorough experiments. That said, I have two major questions regarding the underspecification problem and the task-agnostic nature of the meta adaption policy f. Looking forward to hearing clarification from the authors. 

Update after rebuttal: I read the other reviewers' comments and the author responses to those reviews. Seems to most reviewers had an initially high sentiment and that that sentiment remains the same after the author responses. I appreciate that the authors took steps to clarify the questions the reviewers had. That said, they made very limited changes to the manuscript and some of the limitations that the reviewers raised (having access to ground truth information, underspecification problem related to maximizing rewards) still remain. Therefore right now I would like to keep my recommendation as is, weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper describes a Sim2Real approach based on adapting simulation parameters to match real-world target tasks. The simulation parameters are adapted in order to facilitate training of a policy, which aims to perform well on the real world task, rather than aiming to exactly match the transition dynamics between the simulation and real-world environments. This allows the proposed approach to be used in scenarios where the dynamics cannot be accurately modeled, such as contact-rich manipulation tasks. 
review: The paper is well written and clearly explains its methods and experiments. In particular, the related work section (and especially the extended version of it in the appendix) is comprehensive and covers many similar approaches in the sim2real and SysID area. The authors do a good job of differentiating between the variety of existing methods, and explaining why their method is different. 

The experiments conducted by the authors are thorough and demonstrate their approach well. I appreciate the complexity of the manipulation tasks chosen. While they do not cover a wide range of manipulation tasks, both of the real-robot environments showcase the idea of “irreducible” gaps between sim and real. In a more general sense, it is very cool to see the scooping task solved at all - it seems like this would be a challenging task even for a human operator. 

While the approach described in the paper seems useful and practical for some robotic domains, several questions and concerns arise when reading through the approach and experiment sections. 

1. The task policy is trained at every iteration; is this not an extremely expensive operation? It seems like this could only work for tasks where the policy can be very simply parameterized with a few parameters. What happens if the task policy needs to be more complex, and doesn’t converge for the specific set of sim parameters?
    
    A similar concern is regarding the training of the adaptation policy in phase 1; does it also require the trajectory representations to be relatively low dimensional? From Sec. A3, it seems like all trajectory observations include a relatively small number of points. Would this approach still be feasible for tasks which require a more expressive representation of trajectories? 
    

1. The LearnInTarget baseline presented in the paper seems slightly too simplistic. Wouldn’t it be a more correct comparison to train the task policy in the target domain to convergence, or at least in the same manner as the training of the task policy in the inner step of phase 1? I expected the LearnInTarget baseline to be an “upper bound” policy, demonstrating what the optimal reward would be in the environment. 

1. While BayesSim and its follow-up work were originally trained for well-modeled environment parameters, wouldn’t they still be applicable to the setting described in this paper? A comparison of the proposed approach to BayesSim would be a welcome addition to the experiments in the paper. In general, the paper would benefit from a comparison to one of the adaptive (domain-randomization based?) SysID approaches described in the related work. Baselines include a uniform DR approach and a SysID approach, but not combinations of the two.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While some questions need to be answered by the authors in order for me to be fully convinced that this paper is complete, the approach described in the paper seems sound and the experiments are comprehensive, displaying a successful application of the approach. Therefore, I believe this paper constitutes a meaningful contribution to the robotic learning community, and should probably be accepted to CoRL.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a method for adapting a simulator to enable good task performance in the real world in the context of learning policies via reinforcement learning. The key idea is to find simulation parameters such that the optimal policy in an environment corresponding to those simulation parameters will perform the best in the real world. This is done by formulating a meta-learning problem where a number of target environments are sampled in simulation, and a network is learned to adapt initial simulation parameters so a trained policy can achieve high performance in these target environments. This network is then used to adapt initial simulation parameters to achieve high performance in an arbitrary real world target environment at test time.
review: This paper proposes an interesting idea and is written very clearly. The idea is to find the ideal distribution of simulation parameters to train a policy that will perform well in some OOD target environment, regardless of whether those simulation parameters actually provide an accurate model of the target environment. This is in contrast to domain randomization techniques, which simply assume this distribution of simulation parameters is uniform, and sys ID techniques, which assume this distribution should just be a point estimate where the simulation parameters are set such that the dynamics most closely match reality. 

Strengths: (1) the method seems to be a novel approach for sim to real transfer that achieves strong results compared to relevant prior work; (2) the idea of the method is clearly motivated, and it is clear why in principle it should be able to outperform previous methods

Weaknesses: (1) there is not sufficient discussion of how the task policy is trained for a given set of simulation parameters in the main text, this seems like a sufficiently important part of the method that it should not be in the appendix; (2) the paper would benefit from a concrete and simple motivating example illustrating some specific situations in which the simulation parameters that lead to good task performance may be different from simulation parameters that most closely match the real world. While there are empirical results suggesting this in section 6.3, it would be very helpful to study a simple case where we can exactly compute the simulation parameters that most closely match reality (of course for a simulation in which reality is OOD) and show that the best parameters for training a policy in simulation to achieve high reward in the real environment are actually not those parameters.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I am generally positive on this paper and am leaning towards acceptance. The presented method is clearly novel and it is well motivated why it should outperform prior works on sim to real adaptation. I think the main weakness of the paper currently is the lack of clear motivation about the precise conditions under which one would expect AdaptSim to outperform SysID methods, which I believe could be addressed by adding an experiment with a toy example as discussed above.
review_title: null

Meta-review: The paper describes a method to learn adaptation policies in simulation to maximize their performance in a target environment which can be deployed to the real robot. As opposed to conventional sim2real methods, this approach is focused on the policy performance in the target environment and not necessarily attempts to capture the real dynamics of the system in simulation. 

The reviewers agree that the paper makes a good contribution, is well written and motivated, and presents solid analyzes. The rebuttal helped to clarify specific issues, such as the need for comparisons to SimOpt. Authors are encouraged to include as many suggestions as possible in the final version. In particular, elaborate on the need to access ground truth information and the underspecification problem related to maximizing the rewards.
Decision: Accept (Poster)