Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new approach, which combines offline reinforcement learning with learning in simulation, without the need of training on the target environment. More specifically, they propose to use offline data to learn a distribution over the simulator's parameters and then use the inferred simulator to train an RL policy online. The experiments show that the proposed approach, OTED, performs significantly better than other offline RL, behavior cloning, and domain randomization methods, particularly when learning from non-expert demonstrations as well as in the low-data regime. 
main_review: ### Strengths

Overall, I think this paper could be a valuable contribution at ICLR 2022 since it is one of the first to make a reasonably convincing case for the effectiveness of combining offline and online RL. Moreover, it shows how one can leverage (a limited amount of) prior knowledge about the environment to create an accurate simulator, while inferring from data the parts which are unknown / uncertain. I believe is a suitable setting for many problems of interest. 

The paper is clearly written, technically correct, the proposed approach is novel (as far as I know) and evaluated quite extensively on a variety of tasks, and compared with strong baselines and ablations. 

### Weaknesses

I think the paper could use some improvements and there are some missing experiments / analysis that could make it stronger, as detailed below:

1. It looks like you are missing the results on MiniGrid. Can you please include and describe them in the rebuttal? It would be useful to know how well OTED performs when learning from images in discrete domains, in order to fully evaluate the contribution and the generality of the method across different benchmarks.

2. Is there a reason for which you are not comparing with adaptive domain randomization (ADR; OpenAI et al. 2019) or your DR baseline essentially ADR? Can you please include more details about the DR baseline and explain how it differs from ADR or add the comparisons with ADR?

3. How does performance change with the accuracy of the learned simulator parameters? This could shed light into the robustness of the method with respect to errors in the learned simulators. This could also help better understand the trade-offs between the compute needed to learn the simulator and the performance gains. 

4. I think the paper should lay out more transparently the kinds of assumptions made regarding the simulator. It should also include a discussion regarding deviations from these assumptions such as large number of parameters that need to be inferred or the possibility of having a wrong model of the simulator. How could one discover the parameters that need to be inferred or detect model misspecifications? I think more context could help readers to understand when this method is suitable and when it is not and make it clear that it does use some prior knowledge about MDP. 

5. The paper is also missing a discussion of the limitations of the proposed approach and future directions stemming from these ideas. These are important to place the work in the larger research context and inspire follow-up contributions. 

6. Another concern I have is regarding how the performance (and simulator accuracy) scale with the number (or range) of the parameters you need to learn. For example, if you have less prior knowledge about the simulator, you may need to infer multiple parameters. I think an experiment showing how this scales would provide additional insight. 

7. The related work section seems to be missing some relevant work such as "AWAC: Accelerating Online Reinforcement
Learning with Offline Datasets" by Nair et al. 2021. This paper also uses a combination of online and offline RL (although it assumes limited access to the true simulator), so a discussion of the two is needed. 

### Minor 

1. Do you have an intuition regarding why there is divergence in Figure 2 for gravity in walker and for all params in hopper?
2. Can you bold the model with the best performance in the tables for readability?
3. Please proof read. There are a few typos on pages 1, 6, 8, and likely more. 
 


summary_of_the_review: Given all the above, I am currently leaning towards recommending acceptance. If the authors address the above concerns (adding MiniGrid experiments, ADR comparisons, analysis of how the performance scales with the simulator's accuracy, and a discussion of the method's limitations), I plan to recommend acceptance (unless the other reviewers change my mind or realize I have missed some crucial details).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method, Offline Targeted Environment Design, OTED, for using offline datasets to train ‘realistic’ simulators by minimizing the divergence between the state-action distributions of the offline dataset and in the learned simulator. The authors also introduce methods for selecting learned simulators and policies across simulators, and compare their method against a range of offline RL methods and online RL methods with variations of domain randomization. 
main_review: ### Strengths
- The proposed method for learning simulator parameters seems well motivated and supported by compelling empirical evaluations. 
- Performance comparisons seem thorough and the method seems especially helpful with lower quality demonstrations, since it only uses the suboptimal demonstrations for simulator parametrization and not policy learning.  Overall, the idea of using an offline dataset of suboptimal demonstrations to learn simulator parameters for online learning is interesting. 

### Weaknesses
- The abstract mentions using this method to close the sim2real gap, however it is hard to say how well this will extrapolate to sim2real applications, as this relies on the offline dataset and online learning domains to be identically parametrized. i.e. even if we have an offline dataset of real world demonstrations, if our simulator can’t isolate and identify the correct parameters to adapt (which is likely much harder with the sim2real domain gap compared to the simulated gap here), would the method still work? 
- The design deviation method also seems comparable to OTED with respect to design errors (in Table 2) — are there evaluations on task performance with this scheme? 
- Missing standard deviations to results presented in Tables 1, 2.  

### Questions 
- How sensitive is the parameter learning to the approximate behavior policy learned (i.e. is a reasonably good policy necessary or will a random policy be sufficient for learning simulator parameters)? 

### Additional Feedback
- Would be interesting to include an ‘oracle’ method — online SAC with ground truth parameters to see the gap with OTED-SAC, if any. 
summary_of_the_review: The method is intuitive and it seems like it is capable of learning the chosen simulator parameters well, potentially leading to novel methods in combining system identification and offline RL. However, it is unclear how necessary the more complex sampling method is (compared to design deviation), and applicability to real world system identification problems seems limited. The paper would be strengthened by examining how well the method works in adapting 'realistic' simulators from real world offline data, where there is more likely to be interactions between multiple simulator parameters and the domain gap is larger.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Motivated by the gap between training environments and testing environments, this paper proposes to use offline data to calibrate the simulator then use online RL methods to train generalizable agents. To minimize the divergence between offline data and simulator parameters, the authors train a discriminator. They also propose two cross-validation methods to select the best simulator parameters to increase the performance.


main_review: 1.	Strengths and Weaknesses

\+ The idea of using an offline dataset to calibrate the simulator is intuitive and interesting. This is an alternative way of using the offline datasets to help the training of decision-making agents.

\- The proposed method is relatively heuristic and may have limitations on the novelty. As the authors mentioned, using the policy gradient method to optimize simulator parameters could have a large variance. The proposed solution (picking the best hyperparameter after multiple fittings) in this paper may not be applicable in high-dimensional cases due to the high computational cost.  

\- If I understand correctly, one assumption of the proposed method is that the model of the simulator is correct and only the simulator parameters are inaccurate. If so, this would limit the application of this method in more complex tasks, where we are unable to access or build a correct model. I suspect that the Offline RL method may outperform the proposed method.

\- There are lots of components in multiple stages in the proposed method. However, the analyses of these components are not enough. For example, the authors use a behavior cloning (BC) method as a surrogate model to collect data from the simulator. Why do they select BC methods and why is this a good choice? What if the BC model is not well-trained and makes the learning of the simulator fail?

2.	Questions

(1)	What if we don’t have an accurate model of the simulator? For example, there could be some friction parameters that are not considered in simple simulators. In this case, the performance of the proposed method will always have a gap. In some worst cases, the learned simulator is too simple to fit the offline data and the online agents overfit the simulator. Could the author provide more experiments on an inaccurate simulator model with fewer parameters than the testing environment?

(2)	The joint training of the discriminator and parameter generator could be unstable. For example, a strong discriminator may make the generator fail. How do the authors deal with this problem?

(3)	In equation (3), a hyperparameter $\epsilon$ is introduced and the authors say that it can balance exploitation and safety. What does safety mean here? I assume this paper is not related to safety since this term is only mentioned here.

(4)	What’s the influence of the seed $i$ in Section 4.2? How large is $i$ enough to reduce the high variance? Are there any theoretical or empirical analyses about it?

summary_of_the_review: In general, this paper proposes a very interesting direction, but important analyses and experiments are missing in the current version. I tend to reject this paper before the discussion based on my concerns in the main review. I may raise the score if the authors can provide reasonable answers to my questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an approach to tune a parameterized simulator to produce data samples that mimic a given offline dataset of trajectories from real data. The core technical contributions involve defining a KL divergence to quantify the difference of the simulator data to a given offline dataset, a training algorithm that interleaves simulator tuning and agent training on simulator data, and selection algorithms to choose a hyperparameterization among simulators and a policy among a set trained on a simulator. Evaluations assess how well the training procedure can recover known simulator parameters and the performance of policies trained on a learned simulator to policies trained with offline RL algorithms on the offline data directly. Results show the simulator can accurately recover parameters and is able to obtain similar performance to offline RL algorithms when tuning certain parameters.
main_review: # Strengths

1) Novel approach
	- Offline RL methods have generally focused exclusively on the policy learning process. Shifting focus to the environment parameterization is a valuable change in approach. Previous efforts in this vein have been limited to efforts like domain randomization that do not attempt to learn a good simulator for the target task.
2) Promising simulator fitting results
	- Results are promising for the degree of accuracy in the simulator fits. There is clearly room to improve when allowing fits of many parameters, but the initial results are encouraging.


# Weaknesses

1) Modest policy performance
	- Given the amount of tuning of the simulator and policy selection processes, the results in Table 1 show very modest results. Behavior cloning seems to generally be a viable alternative to the simulator in most cases and is substantially easier to implement and train.
		- Action: Explain why BC would offer such strong performance.
		- Action: Perhaps add an explanation of the additional benefit to having a simulator vs only the final policy.
	- Table 1 does not seem to fully support the claim "We see almost no performance drop when all parameters are trained jointly": both hopper-medium-expert (~20 points lower) and hopper-expert (30 points lower than the next worst) values seem much lower than the single parameter variants.
		- Action: Could this be explained in terms of the environment dynamics? (Or connected to the results in Appendix D with difficulty of learning the hopper environment?)
2) Lack of error bars / statistical tests for policy training results
	- The core results on policy training (Table 1) have no measure of variance or a statistical test of the differences among methods.
	- I recognize the method is computationally intensive to train, but these results are weakened by the lack of any statistical validation of the outcomes. Repeated trials are necessary to establish confidence in the outcomes.
	- I highlight this outcome as it is the core end goal of using simulator tuning: obtaining a better policy for the real task.
	- Action: Add some form of repeated trials and statistical differences test to the policy training results.
3) Lack of statistical tests of differences for the simulator ablations
	- Action: Add repeated trials and statistical tests to show differences (or lack thereof) among the ablations of OTED.

Note: The abstract mentions strong performance with as few as 5 demonstrations. I could not find where these results were reported.

Minor note: I would suggest choosing a different name for the parameter $\epsilon$ as this is typically the symbol used to indicate a noise parameter for sampling distributions (or a very small scalar quantity). For example, $\lambda$ is a widely used name for a trade-off weighting parameter.

summary_of_the_review: The paper presents an important approach to address offline RL training by tuning a parameterized simulator to produce training data for a policy. This has great potential for further investigation and comparison/combination with other offline RL approaches. The empirical results are weak in the current form, leaving me less certain the work is mature enough for publication. I incline toward seeing this as sufficient contribution to merit acceptance.

My score would improve if the authors can provide clear statistical evidence of the superiority of the new method.
review_title: null

Meta-review: This paper proposes a new approach, which combines offline reinforcement learning with learning in simulation. There were different views on the paper among the reviewers and we had quite a lot of discussions. As a consequence, there were still serious concerns remaining, e.g., whether the results are significant enough, whether there are clear advantages of the proposed mehtod over directly using offline RL methods. It is not justified whether the proposed framework can use offline data more efficiently or better reduce the gap between mismatched simulators and offline data. The reviewer who gave the highest score decided not to champion the paper. Considering all the discussions, we believe the paper is not ready for publication at ICIR yet.
Decision: Reject