Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper uses insight from Bayesian optimal experimental design and define an *acquisition function* which quantifies information gain about an MDP's optimal solution given a state-action pair. Such an acquisition function would provide a mechanism for selecting which transitions to query from a ground truth model in model-based RL, in hopes to greatly improve sample efficiency of a learning algorithm. They propose a method for computing this acquisition function, and evaluate it empirically.
main_review: ## Strengths

1. The paper is very well written.

2. The proposed framework and algorithm are well motivated with repeated grounding in practical considerations around problems where a ground truth model exists but is expensive to compute samples from. Independent of empirical evaluation, there are reasonable intuitive arguments for why one might expect improvements in sample complexity.

3. A wide variety of relevant baselines were compared against, including some ablations of the proposed algorithm. The empirical results presented appear promising.

## Weaknesses

1. The current setup they used for BARL appears relatively computationally expensive (e.g., using GPs to model the dynamics, running MPC over every posterior transition function, etc.)

2. The domains are on the simpler end, which are reasonable for the scope of the paper but leave questions about how the approach might scale to larger domains.

3. Only five random seeds were evaluated. Recent criticisms have emphasized how five seeds might not be representative of the data distribution- can the authors comment on the statistical significance of the results? Along those lines, I couldn't find in the text what the shaded regions represent in Figure 3? Can the authors further comment on how hyperparameters were selected across each algorithm and whether they provide a fair comparison?

## Minor stuff which didn't affect the review:

"the the" near the start of Section 3.1
"evaluatation episdoes" near the start of Section 6
summary_of_the_review: Taking the above into account, I recommend acceptance of the paper. While I feel the empirical evaluation could be improved, I find the motivation and intuitive arguments behind the proposed algorithm convincing enough to be a useful contribution in the literature.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the very relevant (in my opinion) problem of data-efficient RL and is potentially applicable in nearly all situations where RL is to be applied in the real-world rather than a simulator. The authors propose an acquisition function based on expected information gain (EIG) approach. The overall framework consists of a (1) Gaussian process model of the transition dynamics, (2) an MPC approximation of the optimal policy, (3) the EIG acquisition function where the target variable is the sequence of states visited by the optimal policy. Empirical results are shown for a number of continuous control domains.
main_review: Strengths:

-The paper provides a principled and interesting approach to an impactful problem. The idea of approximating the optimal policy and then computing information gain on the sequence of states visited by the optimal policy is both novel and clever.

Weaknesses:

-The paper uses independent GPs to model the transition dynamics of the MDP. Could multi-task GPs be used here to improve the model’s capability? For example, [1] allows for  high-dimensional, correlated outputs, which could be used to scale your method to larger problems.

-I’m curious about the sensitivity to the approximation of the optimal policy. For example, such a study could be carried out on a domain where the optimal policy is computable and then various approximations to it (e.g., MPC, MCTS, myopic policies) could be used as the approximation.

-Even though the motivation is for problems with expensive transitions, wall times for the new method should be included so that readers can get a sense for how computationally expensive the method is.

-There are references that might be relevant. Please see [2, 3] from the OR literature.

[1] https://arxiv.org/abs/2106.12997
[2] https://pubsonline.informs.org/doi/abs/10.1287/opre.2018.1772
[3] https://pubsonline.informs.org/doi/abs/10.1287/opre.1100.0873

summary_of_the_review: The paper develops a novel method for an important problem. I, however, have a few comments above that I believe would improve the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an active learning algorithm for model based reinforcement learning. The algorithm is based on a new acquisition function that finds the most informative transition to query an oracle model in order to achieve better performance with respect to the optimal task.
main_review: The main idea of the paper is really interesting. Most model based RL approaches are based on data efficiency, so clearly the idea of applying active learning is interesting. Furthermore, the paper focuses on active learning for policy optimality, instead of the most traditional exploration/exploitation approach.

However, I have trouble following the approach in the TQRL setting that the authors proposed. In that setting, there is already a generative model/oracle/simulator that allows querying arbitrary transitions. If we already have a transition oracle that we can query, what is the point of learning a transition model? Most planning approaches, like the ones presented in the paper and compared against are based on MPC with sampling of the transition model. If the control part was done in a fashion that required the full distribution, I could imagine some possible contribution, but given the use of CEM, it seems that the paper is trying to solve a problem that it is already solved at the beginning. For example, the 3 papers that the paper cites to justify the TQRL setting, one of them is for planning, not RL, and the other 2 are about theoretical analysis on sample complexity. There is not a practical implementation in an RL setting.

Being familiar with the RL and BO literature, the paper can be hard to follow. For example, the acquisition function is presented (s4) before the context where it is applied (s5). Also, in the Algorithm it is hard to find where the reward is optimized until the reader realizes that the 3rd line in the loop (executing pi) implies computing the optimal policy by MPC/CEM and then, generating the trajectories. It is not just executing a policy. The acquisition function also needs more clarification: the authors choose to compute O instead of a fully defined pi* to avoid sampling unnecessary states that are never visited from p0. This affirmation implies that both the initial distribution and the transition distributions are considerably limited. If the initial state is unknown or the noise level is reasonably high, the advantage of using O vanishes, because the trajectories end up visiting most of the state space. But the choice of O might have some limitations: what happens if the initial policy/trajectories are completely wrong, driving the queries to irrelevant state/actions for the goal? Is there any convergence guarantee?

The state of the art seems to miss an important part of literature on intrinsic motivation, curiosity, which have addressed similar problems before. I recommend the works of Pierre-Yves Oudeyer, Jürgen Schmidhuber, etc. For example, the work on Lopes et al. seems very related to the paper at hand.

Lopes M, Lang T, Toussaint M, Oudeyer PY. Exploration in model-based reinforcement learning by empirically estimating learning progress. InNeural Information Processing Systems (NIPS) 2012 Dec 3.

Furthermore, given that the surrogate model in this paper is based on GPs, there should be a mention and comparison to GP-based model based RL methods, such as PILCO, which has been, and they still are, the standard of sample efficient RL. 

Deisenroth MP, Fox D, Rasmussen CE. Gaussian processes for data-efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence. 2013 Nov 4;37(2):408-23.

In fact, the MPC methods that the authors include in the comparison, seems to be a stripped down implementation of this work.

Kamthe S, Deisenroth M. Data-efficient reinforcement learning with probabilistic model predictive control. InInternational conference on artificial intelligence and statistics 2018 Mar 31 (pp. 1701-1710). PMLR.

In contrast, it seems that the inclusion of model-free methods in the comparison just adds noise to the plots. Maybe you can include the performance in the table and supplementary material and, instead, include GP-based RL in the plots, like those mentioned before.

Minor details:

-For fair comparison, given that the competitors cannot exploit the transition oracle, it would be interesting to see performance BARL in environmental steps, including the steps needed to reach the desired query. That is, the algorithm is able to sample arbitrarily any transition, but the agent has to reach it sequentially.

-The most complex problem in terms of exploration space seems to be the reacher which is 8D. How does the algorithm behave in higher dimensional problems, such as half-cheeta, ant, humanoid, etc. Or even some exploration challenging scenarios such as the ant-maze.

-The authors use EIGt as a proxy for the exploration algorithms. Why not use the full algorithms instead? For example, the code of Shyam et al is available.

-What is the implementation of the cartpole swing-up? Why haven't the authors used a standard benchmark such as the OpenAI gym version with mujoco or pybullet?

-Second paragraph, third line -> repeated “on”

Updated review: 
----
After reading the updated version of the paper, I see the relevance of the proposed method and the TQRL setup: The paper is now much more readable and I have updated my score accordingly.

My only remaining major concern is with the comparison. Being a novel setup I understand the complexity of a fair comparison and the addition of PILCO clearly solves some of the questions. However, there are still some questionable choices. For example, in the submitted code, there are multiple versions of the cartpole and it is unclear which one is used, and the fact that there are some versions with the names of specific methods could be interpreted as different versions where used (which I assume was not the case). It's true that the OpenAI gym cartpole is not swing-up, but there is one in pybullet and there are many well known benchmarks such as dm_control. Using previously defined, unaltered tasks, like the Pendulum and Reacher would make all the experiments much more stronger.
summary_of_the_review: The main idea of the paper is interesting and relevant for the field. However, I feel that the setting and the comparison needs to be revisited to have practical application.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an active learning method for transition query RL (TQRL) in continous domains. In TQRL, an agent queries the dynamics of the transition function at arbitrary state-action pairs in order to estimate the system dynamics. The objective is to query pairs that are most useful in solving the underlying control problem. The paper formulates TQRL as an expected information gain (EIG) maximization given the current dataset of observed state transitions, and applies posterior function sampling to evaluate EIG. The method is empirically evaluated in five continuous control tasks, with improvements in empirical sample complexity.
main_review: Strengths:
- Simple and easy to understand method for active learning in TQRL, that is based on the principle of expected information gain (EIG) from Bayesian experimental design. While not optimal, similar greedy maximization approaches have a strong track record in related problems such as active information gathering.
- A nice reformulation of the problem as reducing uncertainty about the optimal trajectory, which leads to a decomposition according to the sources of uncertainty.
- Empirically demonstrated to reach good performance with only a handful of samples, outperforming model-free methods and in some cases model-based competitors such as PETS.

Weaknesses:
1. The empirical evaluation only concerns simple domains where a Gaussian process (GP) model easily suffices. It is not clear if the methodology will scale to larger or more complex models, where evaluation of the EIG objective function itself might pose challenges.
2. To the best of my understanding, the performance of the proposed method is also capped to the performance of an MPC solution in a problem. This also shows in the results, e.g., Fig. 3 showing PETS on beta tracking and reacher obtaining a higher final reward, SAC & TD3 in other domains. While this is not necessarily an issue in the domains considered in the paper, generally MPC cannot be guaranteed to reach the same performance as an optimal solution. This is a limitation of the proposed approach.
3. There are a few points that need additional clarification, details below.


Detailed comments:


Questions on Algorithm 1:
- Why is # of points for optimization k an input when k does not appear in the algorithm?
- Line 2: does U refer to uniform distribution? In the initialization, should s_0 be sampled from p_0?
- Line 7: I don't understand why X is drawn from an uniform distribution over S x A. Then, X would be a pair of a state and an action. This does not really make sense for how it is used on Line 8 as a set.

Presentation of the results:
- Fig 1 (b), the colored trajectories are distracting, are they necessary?
- The significance of the colors in Fig. 2 is not clear.
- Could you clarify what exactly is the MPC method shown in Table 1, Fig. 3? I did not get it from the paragraph "Comparison Methods".



Other/Minor:
- This comment is just for information and needs no response: Considering the GP models, submodularity arguments might be helpful to further justify the greedy sampling approach. See, e.g., Krause et al. "Near-optimal sensor placements in Gaussian processes: Theory, efficient algorithms and empirical studies", JMLR 2008. 
- Just before Eq. (1), there range 1, ..., H for the index i has a mismatch. That is, a_0 is not accounted for and no action should be drawn at step H.
- "The target the minimal information..." -- sentence seems ill-formed, please revise
- Eqns. (3)-(4): to remove any guessing, please mention here that blackboard H is the entropy
- "Given sufficient computational resources and an accurate model, model-predictive control will typically find very good if not optimal actions in an MDP." -- could you point to a source for this claim and/or make precise what this means? AFAIK, no error bound for the performance of MPC compared to an optimal solution is known.

**Added after the author response**
I thank the authors for their response, which clarifies my remaining concerns. I have increased the score to reflect this.
summary_of_the_review: This paper proposes a simple but seemingly effective method for model-based RL in cases where obtaining transitions for training from the real environment is extremely costly. The method is based on well-founded principles of Bayesian experimental design. The paper itself is clearly written and easy to understand, and technically sound. The empirical results sufficiently clearly show the advantages of the proposed methods. More realistic experimental settings could be considered in future work.
review_title: null

Meta-review: In this paper, the authors introduce an exploration method for RL according to experimental design perspective via designing an acquisition function, which quantifies how much information a state-action pair would provide about the optimal solution to a Markov decision process, and the state-action that maximizes such acquisition function will be used for sampling for policy update. The empirical evidences show the proposed method is promising. 

Since most of the reviewers support the paper, I recommend acceptance of this submission. 

However, besides the questions raised by the reviewers, e.g., computation cost and planning quality from CEM, there is a major issue need to be clarified in the paper:

>The algorithm designed for RL with generative model, which makes the state-action reset can be conducted (this is sometimes impossible in practice where the agent must start from initial state). This is different from the common RL setting, and thus reduce the complexity of RL. This should be emphasized in the paper. Meanwhile, for a fair comparison, this should be explicitly specified in experiment setting.
Decision: Accept (Poster)