Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn neural environment models they term Synthetic Environments, such that planning in the model with model-free RL results in good performance in the real environment.
The authors study learning full models as well as reward-only models they call Reward Networks, investigating the feasibility of learning these models as well as their robustness to different inner-loop algorithms and hyperparameters.
main_review: This is an interesting line of work and initial investigation. The exposition of the method and description of experiments are quite clear and complete. However, there are a few weaknesses.

The motivation for this work deserves more attention. The authors suggest that SE’s could be used for AutoML, iterating on algorithms in expensive environments like robotics, agent or task analysis, or agent pre-training. However, I’m not sure these are particularly well aligned with the authors’ statement of the problem, implementation, or experiments.
Most applications are predicated on the relative performance of various algorithms, when trained in the SE, being correlated with performance in the true environment.
However, the problem formulation in eqn (1) describes maximising performance absolutely.
This is approximated in Alg. 1 with a generic TrainAgent function that approximates the argmax_theta. But it seems the specific TrainAgent functions chosen are highly relevant to potential use cases – it is not a generic stand-in for an argmax.
The authors certainly understand the relevance of this, as they study generalisation of SE’s to different algorithms and their hyperparameters, and even address generalisation algorithmically by varying hyperparameters during training of the SEs. However, this is introduced as a brief comment in the experiment descriptions, rather than being central to the method.
It feels that to use SE’s for the potential applications discussed, it would make more sense to reformulate the problem statement to explicitly account for different inner-loop algorithms.
Further, simply maximising the final performance of all inner-loop algorithms seems quite limited when applications involve comparing different algorithms.
It might be interesting to algorithmically address this use-case by changing the fitness function to account for the relative performance of different algorithms.
However, it would also be useful to address this empirically given the current formulation of the objective, and the authors likely already have the data to do so, at least partially: does the performance of different HPs/algorithms *in the SEs* correlate with their performance on the true environment?

It may also be possible to motivate the work from a more traditional model-based RL point of view, which might be more closely aligned to the problem statement given in the paper. A closely related work is that of Nikishin et al [1] (who use implicit differentiation to address the bi-level optimisation). NB I don’t fault the authors for not citing this, I believe it is only available as preprint. They argue that models learned only to further optimisation of a policy (as in this work's problem statement) may perform better than traditional models when capacity is limited. However, the ES-based bi-level optimisation is expensive, so it may be difficult to make a strong case for this approach.

Another weakness of the study is the simplicity of the environments used. I understand the authors may have limited computational resources, but it is difficult to know how highly to weight an empirical feasibility study carried out on such small-scale tasks. In the absence of more compelling empirical findings, it is more critical to explore the algorithmic choices and motivations.

In this vein, I did appreciate the investigation of which inductive biases might help during reward learning. Overall, I find the case for RNs is perhaps more intuitive: dense-ifying rewards once in an expensive optimisation to allow fast iteration later. Using an additive potential form even guarantees that this will not change the optimal behaviour – sadly, it seems this type of RN did not outperform the baseline much in the experiments.

In the RN case optimising for the speed of training makes sense, but I still find the chosen objective strange. Why not optimise the area-under-curve returns for the full inner-loop? Clipping at a somewhat arbitrary solution threshold seems like it would result in suboptimal RNs (this appears to have occurred in the half-cheetah experiment in the appendix).

Minor comments:
 - In the SE experiments, it would be good to show the performance of the ‘fixed’ hyperparams evaluated with the same fixed hyperparams: this would show the ‘generalisation gap’ compared to evaluating with varied hyperparms as is currently shown.
 - Overall, many figures are a bit hard to parse. Try using more smoothing or subsampling for curves. For the barplots, maybe re-arrange them so the comparable bars (i.e. measuring the same metric, ‘steps’ or ‘episodes’, are next to each other).

[1] “Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation” Nikishin et al 2021

----------------------

Update: the authors have addressed many of my concerns, ran some additional useful experiments, and clarified their reasoning on several points in the overall discussion with reviewers. I still think there is a bit of a gap between the big-picture motivation for the work and the instantiation we see studied empirically; the authors are working on an empirical investigation to help make that connection. However, even in its absence, I am sufficiently positive about the paper overall now to lean towards acceptance -- I found the study interesting and creative.


summary_of_the_review: While the work is interesting, I find the motivation a little strained at not well aligned with the problem statement or algorithmic instantiation. The empirical study is fairly clear but limited by the simplicity of the domains and the restriction largely to assessing feasibility rather than giving clearer indications of how the method could be used in practice.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a framework for learning to synthesize a proxy models for usually non-learned components of an RL loop: namely transition dynamics and reward as defined in the interacting environment. They learn the parameters of these model by "meta-learning" onto the real environment with a learning agent (with slight detail variants depending on what the learning objective is), showing that not only can this setup lead to successfully learn the proxy parameters, but also improve learning performance for the agents.

main_review: 
First and foremost, let me start by saying that I *really* like this manuscript.

1. The writing is extremely clear & detailed, and makes the paper a pleasure to read. I also found that the authors tended to answer questions that were popping as I was reading paragraphs immediately thereafter, which is a likely sign of writing that has gone through a lot of careful iterations.

2. The authors have gone through a lot of work to justify the choice of optimization methods, how and why the framework was setup in certain ways, and how the hyperparameters for a now fairly complex learning system were chosen. Considering the code release and the extremely detailed appendices, I suspect the work will be extremely easy to reproduce. Truly commendable, and a great example of what good ML research looks like (I also particularly enjoyed the frank discussion about the work's limitation due to choosing ES as an optimization method for the outer loop).

3. The central idea of the method is relatively simple, but seems to be quite effective; it honestly borrows from a lot of previous literature (reward learning, population-based optimization, etc.), whilst effectively producing a novel take on learning syntethic environment models.

So, overall, I am extremely happy to argue for acceptance as it is. Now, let's look at what I perceive to be some of its weaknesses:

4. The manuscript at times feels very dense, especially when looking at methodology and experimental details, and its understanding is greatly helped from the presence of a sizable appendix. It very much feels like the authors could have a feasibly split the paper into two, one on ES and a follow up on RN, providing more space to experimentally analyse ES and RN separately and make for a cleaner and easier review process.

5. Some of the hypotheses presented to justify the improved agent training feel relatively handwave-y. Consider for instance the following quote (emphasis mine):

>Optimizing SEs for the maximum cumulative reward automatically yielded fast training of agents as a side-product, likely due to **efficient synthetic state dynamics**.

It is implied that this meta-learning approach leads to generally discover better data-producing dynamics for the agents used in the system, but I wonder whether this is primarily due to the structure of the tasks employed in the experimental settings (as state space and transition dynamics of these simple control problems are _generally_ not very informative, and the _useful_ -- policy learning wise -- parts of these MDP are instead both easily discoverable via search and extremely informative). 

In short, I wonder if we'd see similar improvements for tasks / environments where task difficulty is extremely affected by the emerging complexity of the environment, or whether we can actually show that this is a property of SEs / RNs when trained in this manner (which would be quite outstanding!). Although that said, the effecitveness of RNs do seem to indeed point to the latter hypothesis.


### Nits / Additional comments

Figure 2: the labels are inverted (dotted lines should be the thresholds).

Section 6 (but generally in many parts of the paper): it is claimed that learning rewards is generally easier than learning transition functions -- would it be possible to find a reference for this? My personal opinion is that it should indeed be often the case, but that generally it is not true (e.g. imagine an MDP with a small transition matrix but an extremely stochastic reward function).
summary_of_the_review: 
This is a good paper. It contains excellent writing, good research, and makes for a great example of what an ICLR paper should look like. The research problem and proposed methods are interesting and well placed in the literature, and the experimental section is exhaustive.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to learn proxy environments (synthetic environments or SEs) and reward functions (reward networks or RNs), parameterized as neural networks, such that these proxy models provide beneficial transitions to make it more sample-efficient to learn a policy for a fixed target environment (referred to as the real environment). SEs replace the observed state and rewards during training, while reward networks provide a synthetic reward that is a function of the real reward, current state, and next state. The proposed method formulates this problem as a bi-level optimization, where the inner loop consists of standard RL under the proxy model, and the outer loop consists of NES with the aim of optimizing either the performance on the true, target environment (SEs) or a the number of training steps needed to reach a certain return threshold on the real environment. 
main_review: ### Strengths
- This paper extends prior work on meta-learning MDP transition dynamics for improving training performance on a target environment, by introducing a method that aims to learn both the state and reward transitions. This formulation is more general than prior works.
- Experimental results are provided for multiple formulations of meta-learned MDP transition dynamics, spanning both joint state and reward networks (SEs) as well as various formulations of a reward network (potential and non-potential-based reward shaping parameterizations).
- This paper includes experiments on transferring to different agent architectures and hyperparameter settings, which seems to be a novel experimental setting for meta-learning MDP dynamics.

### Weaknesses
- The motivation for learning SEs and RNs for singleton target environments is not convincing. While there seem to be some marginal gains in sample-efficiency, it would seem these gains are largely made irrelevant due to the additional training needed to train the environment in the first place. Moreover, if SEs and RNs require first training the agent multiple times on synthetic environments, why not just copy weights or use a form of policy distillation, e.g. kickstarting (Schmitt et al, 2018), rather than train the agent again? Further, it seems the basic ICM baseline already matches or outperforms the proposed RN approaches across experimental settings, further weakening the argument for using the more computationally expensive method proposed in this work.
- Given the wide design space of possible ways of learning the state and reward transition dynamics for a target real environment, it is unclear why only these two (SE and RN) formulations were studied. Given that this is an empirical paper, the value of the findings rests largely on a clean experimental design that provides a convincing recommendation based on comprehensive experiments. The seemingly arbitrary choice for only considering the SE and RN formulation does not provide such a convincing recommendation for using either of these formulations.
- RNs are effectively an ablation of SEs, where the state is not learned. However, the current experimental design does not neatly present these results as a clean ablation. This is because the RN experiments also separately consider various kinds of potential and non-potential-based reward shaping parameterizations for the learned reward. These parameterizations are not studied in combination with SEs.
- Likewise, there is no corresponding ablation for when the reward is not learned, but only the state is learned, i.e. a "state-transition network" or perhaps a "state-augmentation network."
- It is not made clear why exactly SEs and RNs have to use different objectives when training.
- The RN objective seems strange, as it seems to be optimizing for sample efficiency rather than performance. In contrast the SE objective optimizes for performance. This seems like a significant inconsistency between the training methology for the two types of models, making it hard to compare the relative merits of SEs and RNs.
- The reward difference term of the RN objective also caps the maximum performance improvement compared to training on the real environment. The motivation for this objective design is not clear, as not taking the max between the reward difference and 0 should encourage learning RNs that lead to outperforming training on the real environment, rather than simply matching it.
- Further, it seems that there is not a consistent ranking in terms of which parameterization for the RNs improves training performance on the real environment the most, and similarly, no consistent ranking between non-transfer and transfer scenarios for each environment.
- Meanwhile, the SE results on CartPole and Acrobot would benefit from a direct comparison to the training curves for agents using the best hyperparameters, trained on the real environment. This would directly separate out the performance gains due to training on the  SE.
- The comparison to previous work such as Zheng et al, 2020 seems quite hand-wavey. Zheng et al, 2020 in particular is quite similar to this work in terms of analyzing the effect of learning an RN and also includes more comprehensive experimental results in a more complex setting requiring recurrent RNs. The novel contributions in relation to these prior works seems to be around analyzing the impact of varying agent hyperparameters on transferring to alternative agent settings and architectures during evaluation. However, these transfer results do not seem consistently strong for any one method proposed.
- Sampling agent hyperparameter settings from a diverse range seems crucial for the learned SEs and RNs to transfer to other agent configurations. The paper does not discuss how sensitive the proposed methods are to the agent hyperparameter ranges used during training.
summary_of_the_review: While this work proposes an interesting generalization of prior work on meta-learning MDP dynamics, the case made for these methods is weak, and the experimental results—especially for SEs—are uncompelling. Given the additional compute required by these methods, it seems that various forms of policy distillation or even training from scratch seem more efficient. Further, it is not clear why only certain formulations (SEs and RNs) of meta-learning neural transition dynamics were investigated, while others (like a state-only neural transition network) were not. Given these points, I recommend this paper in its current form for rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Synthetic Environments and Synthetic Rewards as a bi-level optimization problem. The main idea is to find SEs and REs such that an RL agent trained under them will do well in a given, known target environment, without further training. 
main_review: Results:
- In Figure 2, the performance seems very unstable: In particular, the reward for individual agents seems to fluctuate widely and even the average starts drooping after the 50th iteration in the arcobot task. It would be great to know what the source of this instability is. 

Conceptual:
- There are repeated claims in the paper that the synthetic environment "focuses the learning" on the relevant states or learns "representations" of the target environment ("an informed representation of the target environment by
modifying the state distributions to bias agents towards relevant states"). This claim is entirely unclear to me at the moment. In particular, there simply doesn't seem to be any reason that the SE should have much in common with the ground truth environment. The only condition is that _optimal policies_ for the two environments match. In particular, there will be many environments in which the optimal policy is likely much easier to learn than in the original Env. 
For example, you can imagine a state transition dynamic that is entirely random but provides a large reward for the optimal action in each of the given states. Another option is to learn environment dynamics where the optimal action in each state deterministically transitions to an arbitrary next state, such that the agent will see all required states during training. 
It would be great to provide any type of evidence that there is actually semantic meaning in the SE that matches the underlying ground truth environment.  

Other comments:
- In Figure 2, it seems like the legend is switched? I.e. dashed line should be the solved threshold?
summary_of_the_review: A very interesting paper with early results in a potentially interesting direction. It is currently unclear what this method allows to do that was previously impossible from a practical point of view, but there seems to be promise. 
I'd be open to raising my score if my concerns are addressed by the authors. 



Update: 
Based on the discussion phase and the rebuttal I have updated the score to a 6. I believe this paper has enough merit to be published at ICLR, if there is space. It is an intriguing piece of work, even though the practical utility is very unclear at this point.
review_title: null

Meta-review: This paper proposes a new method for generating synthetic environments and reward networks for reinforcement learning tasks. This happens as a nested process: policies are learned in an inner loop, and environments are evolved in an outer loop. The environment representation is quite simple: the parameters of an MDP. Similarly, the reward networks are simply neural networks. Results show that the the learned environments and reward networks are reasonably good at decreasing policy training time by RL. The proposed method appears to be simple and quite general, and it would be interesting to see how it scales up to more complex environment representations.

The discussion around the paper centered on understanding various details of the method, and on the quality of the results. The reviewers generally agree that the paper is easy to read, and vary in their assessment of the significance of the results. It was pointed out that the generated environments are not necessarily similar to the base tasks, but it was nowhere claimed in the paper that they were. (In fact, it could be argued that the dissimilarity makes the method more interesting, given the good results of policy training.)

I'm happy to recommend the paper for poster acceptance. If the results would have been more impressive, it could have been accepted for a more prominent presentation form; however, I believe that the method can yield better results in the future with more sophisticated environment representations.
Decision: Accept (Poster)