Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
This paper proposes to adapt RL agents from some set of training environments (which, in the current instantiation, vary in some simple respect) to a new domain. They build on a framework for model-based RL called PETS. 

The approach goes as follows: 

2-step process
 * train probabilistic model-based RL agents in a “population of source domains”
 * dropped into new environment use “pessimistic exploration policy”

Then at test time, in order to compute estimates for the rewards for each action the authors use a “particle propagation” technique for unrolling through their dynamics model .

The action is chosen by looking at the sum of the 0 through kth percentile rewards. 
This is a weird choice. Why are they looking at a sum over quantiles vs a quantile itself?

The claim is that the models from the first stage capture the epistemic uncertainty due to not knowing z.
However, the authors give a too scant a treatment of what these uncertainty estimates really mean.
For example, they appear to only be valid with respect to an assumed distribution over z.
The paper’s experiments however focus in large part on what happens when the model is evaluated 
on values of z that were outside the support of the distribution over training domains. 
In this case, any benefit appears to be ill explained by the underlying motivation.


The next step here is to finetune the model as data is collected on the new domain.

Authors propose heuristics for this finetuning that include
1. Drawing experiences from the past experiences (under different domains) and 
2. “keeping the model close to the original model”, via some sort of regularization presumably.

>>> 	why isn’t the exact nature of how they “keep the model near the original model explained in the text?
	perhaps the authors mean that 1. and 2. are one and the same (1 as  means to achieve 2)
	if this is the case, then the exposition should be improved to make this more clear.


Some important details appear to be missing. For example, how many distinct source domains are seen during pretraining? Do they set z different z for every single episode of pretraining? Some language here is unclear, for example what precisely does an “iteration” mean in the context of the experiments? 

The choice to report “average maximum reward” seems strange if what the authors care about is avoiding risk. Can they explain/justify this choice or if not, present a much more comprehensive set of experimental results?

The figures tracking catastrophic failures vs performance resembles those in 
“Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear”  https://arxiv.org/abs/1611.01211
This raises some question about why they don’t if concerned with “catastrophic events” model them more explicitly. 
Else, if the return accurately captures all desiderata, why to we need to count the failures?

In short this is a simpzle empirical paper that makes use of heuristic uncertainty estimates, 
including in settings when the estimates have no validity. The writing is reasonably clear
and the ideas are straightforward (which is perfectly fine!). A few of the decisions are unnatural,
a few are ad hoc, and a few details are missing. Overall my sense is that this paper 
has some good qualitities, including the clarity of much of the exposition, 
but it’s still below the mark to be an impactful ICLR paper. 

==========UPDATE=================
I read the rebuttal and am glad that the authors took time to read my review and engage with the criticism as well as try to make some small improvements to the paper, especially exploring the impact on the number of training environments on the results (in the original paper the number of environments available at train time was unlimited). The answers to some of the other questions were less convincing. E.g. the seemingly incoherent objective of summing over the quantiles falls flat. Why should we care more about being a "strict generalization" of some previous algorithm built upon than of having a coherent objective? Overall, I don't think the paper makes it over the bar to accept but I hope the authors continue to improve upon the work and get it into shape where it could be accepted at another strong conference.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper tries to address the safe adaptation: given a model trained on a variety of past experiences for some task, train a model learning to perform that task in a new situation while avoiding catastrophic failure. 


Pros:
- The idea of training on data from varying quartiles, with the goal of preventing overly-conservative models, is quite intriguing and inspiring.

Cons & Question:
- Motivation:  
Cautious exploration or optimizing the best worst-case performance is conflicting with the philosophy of exploration, such as UCB. As stated in the introduction, “enables fast yet safe adaptation within only a handful of episodes.” Intuitively, we can not expect to be safe and fast at the same time. It would be better to discuss why cautious exploration can ensure fast and safe adaption, which would be more interesting. Additionally, in Figure 3, some fast adaption methods, such as MAML, should be compared to be more persuasive.

- Method:
 1. In equation (1), sum_N —> sum_i.
 2. This work formulated safe adaption as minimizing the risk of catastrophic failure. What’s the relationship between “the generalized action score” and “risk of catastrophic failure”? The “generalized action score” is the main difference with PETs. However, it is a little bit hard to follow the idea from “risk of catastrophic failure” to “the generalized action scores”. 
 3. “Model-based RL agents contain dynamics models that can be trained in the absence of any rewards or supervision.” 
 ”Since dynamics models do not need any manually specified reward function during training, the ensemble model can continue to be trained in the same way as during the pretraining phase.” I am confused about these sentences. Without the reward, what’s the purpose of RL? 


- Experiments:
1. As stated in experiments, three meta-learning approaches have been deployed as baselines, including GrBal, RL^2 and MOLe. However, the experimental results are missing. Why meta-learning baselines do not work? Are there any explanations?
2. There are many robust RL baselines, such as 
 [1] Pinto L, Davidson J, Sukthankar R, et al. Robust adversarial reinforcement learning[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017: 2817-2826. 
It would be better to compare with robust reinforcement learning work since there are no other baselines apart from meta-learning methods.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of safe adaptation to avoid catastrophic failure in a new environment. It draws intuition from human behavior. The proposed method (risk-averse domain adaptation (RADA)) learns probabilistic model-based RL agents from source domains, and uses them to select actions that has the best worst-case performance in the target domain.

The paper mentions safety-critical applications like auto-driving. However, generally, I don't think black-box models are suitable for these safety-critical applications.
review_title: Official Blind Review #2

Meta-review: The work this paper presents is interesting, but it is not quite ready yet for publication at ICLR. Specifically, the motivation of particular choices could be better, such as summing over quantiles, as indicated by Reviewer 1. The inherent trade-off between safety and speed of adaptation and how this relates to the proposed method could also use a clearer exposition.
Decision: Reject