Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method for exploration in MBRL techniques by proposing a method called trajectory information planning. They present an algorithm to maximize the expected information gain for the trajectory during planning. The method have been demonstrated on several control problems and compared with some baselines.
strengths_and_weaknesses: Strengths:
1. The information gain metric is interesting and seems novel to me.
2. The authors can show improvement over some chosen baseline methods
Weaknesses:
1. I think the evaluation of the paper needs more work.
2. Would it be possible for authors to perform comparison on some high-dimensional problems? Also, I think it is unfair to compare to model-free methods as they are known to slow compared to model-based method on these classical control tasks. If you would like to compare your method to model-free methods, please select some tasks where the SOTA MBRL techniques dont perform good compared to model-free methods, and try to show that your method succeeds. Otherwise the comparison seems unfair.
questions: 1. How would your method work for high dimensional tasks? How would you method compare against a Stochastic MPC approach? like chance constrained MPC? or Tube MPC kind of approaches. In general, they tend to work well on classical control systems.
2. How would your method scale to high-dimensional systems? Did you try implementing your method on acrobot or humanoid kind of systems?
limitations: 1. I think authors can try and address some limitations with respect to computations and scaling of the method to high dimensions. That could be useful insight for the readers.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper extends Bayesian Active Reinforcement Learning (BARL) [38] by lifting its Transition Query RL (TQRL) assumption (TQRL means that the agent can sample individual transitions $(s,a,r,s')$ by querying the simulator at an arbitrary $(s,a)$ pair). Instead, only state-action pairs from collected trajectories are used in this paper.

The main contribution is Algorithm 2 for computing the cost function $C_{\tau^*}(S)$, the negative joint expected information gain about the optimal trajectory. The average cost is subsequently optimized via the cross-entropy method (CEM). The computation complexity of one iteration is derived, and experiments on 5 environemnts for closed-loop policies (the algorithm applied in MPC fashion) and 3 environments for open-loop policies are reported.

The proposed method Trajectory Information Planning (TIP) is shown to perform comparably or even better than BARL in the sense of requiring fewer samples. The fact that TIP is even better than BARL is surprising since BARL can sample the state-action space more freely, but the authors hypothesize that this is because BARL samples uniformly whereas TIP has a more sophisticated scheme and forces initialization at the start distribution.

To demonstrate that explicit exploration is useful, TIP is shown to require fewer samples than the methods which don't explicitly encourage exploration (MPC, PETS, SAC, TD3, PPO).

Furthermore, TIP is compared to a few baselines — HUCRL and TS — that rely on upper confidence bound (UCB) and Thomson sampling (TS) over the dynamics and shown to provide better exploration by requiring significantly fewer samples.

Finally, a few ablations of the proposed method are considered that either only use the cost function $C_g$ (used in MPC) or the entropy objective $C_e$ (used in DIP, EIG$_T$). A variant of TIP where the information gain is split over time steps called sTIP is also considered. All these methods perform comparably, according to Table 1, however TIP shows the best sample complexity.
strengths_and_weaknesses: Strengths
- clarity: well-written paper, with good structure and clear presentation

Weaknesses
1) Significance: medium to low. Too dramatic and misleading claims in the abstract
    - "2-200x fewer samples compared to 14 baselines" — the 200x improvemnt is w.r.t. methods that don't compute information gain (PETS/SAC/TD3/PPO). Such comparison does not seem fair. There has been a lot of algorithms that propose various version of information gain and artifical curiosity, therefore a comparison should be with respect to those algorithms or at least w.r.t. BARL or MPC with entropy objective. Compared to these algorithms, the contribution is still valuable but the statement would be more sober.
2) Originality: fair
    - Reading the paper briefly makes an impression that the whole approach of minimizing the information gain about the optimal trajectory is new. In particular, this is also claimed in the abstract. However, all the derivations and ideas are already presented in BARL [1]. The main contribution of this paper compared to BARL is to compute the InfGain using only sampled trajectories. It should be made much more clear what is the contribution of the present paper and what is actually novel here.
3) Quality: fair. A lot of evaluations but comparison to wrong baselines
    - As mentioned above, the baselines are either from a different category (SAC/TD3/PPO) or don't use InfGain (MPC/PETS) or seem to be badly tuned or not applicable (HUCRL/TS) or are version of the proposed algorithm (sTIP, DIP, EIG$_T$). On the other hand, a variety of exploration bonuses and surrogates for the InfGain have been proposed, e.g., [2,3]. In particular, the proposed method appears very similar to [4] and [5] and therefore either a comparison would be desirable or a statement why such comparison is not necessary.


References \
[1] Mehta, V., Paria, B., Schneider, J., Ermon, S., & Neiswanger, W. (2021, September). An Experimental Design Perspective on Model-Based Reinforcement Learning. In International Conference on Learning Representations. \
[2] Shyam, P., Jaśkowski, W., & Gomez, F. (2019, May). Model-based active exploration. In International conference on machine learning (pp. 5779-5788). PMLR. \
[3] Pathak, D., Gandhi, D., & Gupta, A. (2019, May). Self-supervised exploration via disagreement. In International conference on machine learning (pp. 5062-5071). PMLR. \
[4] Schultheis, M., Belousov, B., Abdulsamad, H., & Peters, J. (2020, May). Receding horizon curiosity. In Conference on robot learning (pp. 1278-1288). PMLR. \
[5] Tschantz, A., Millidge, B., Seth, A. K., & Buckley, C. L. (2020). Reinforcement learning through active inference. arXiv preprint arXiv:2002.12636.
questions: 1) In Fig. 3, "Model MSE on Random Set" is bigger than "Model MSE on Current MPC" by 4 orders of magnitude. This seems to imply that the model is quite bad outside of the observed data. How should one interpret it? Can one then rely on such model?
2) Table 1 shows only the median value over 5 runs. It would be usefull to see the whole distributions, at least in the appendix. E.g., display a boxplot for each environment. Otherwise, it is hard to judge how reliable the numbers in Table 1 are and what is the variance in these results.
3) MPC seems to perform quite comparably to TIP and it requires significanlty fewer samples than SAC/TD3/PPO/HUCRL/TS, despite the fact that it has no notion of exploration and is only optimizing the cost function $C_g$. This seems to indicate that explicit exploration may not be so necessary in the considered tasks or that MPC somehow performs sufficient exploration anyways. It would be worth providing a discussion of this result.
4) DIP appears very similar to MPC in Table 1 despite the fact that they are using completely different objective functions. DIP is only using the entropy objective $C_e$ whereas MPC is only maximizing the reward  $C_g$. Such similarity seems to indicate that the reward is somehow correlated with the exploration in the considered tasks. Or is there a different explanation?
5) Results for BARL are better in the BARL paper (Table 1 here https://arxiv.org/pdf/2112.05244.pdf) compared to the results in Table 1 in the present paper. What is the reason for that?

limitations: Technical limitations are described sufficiently. There is no discussion of the negative societal impact. The authors could indicate potential dangers of active exploration in applications of RL.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper aims at improving data efficiency for RL, targeting real systems where data acquisition is normally costly.
The authors consider model-based control for MDPs with explicitly modelled dynamics.
Control is relegated to MPC, the procurement of parametric (amortized) policies is left out of scope.
The focus is on obtaining useful data for learning the dynamics model as fast as possible, so that MPC can successfully control the agent.
The method shows significant improvement in data efficiency in 5 different environments with simple to moderately complex dynamics.

The main premise is that explorative actions are selected such that maximum information (in the sense of infogain, mutual information, MI) is gained for the optimal future trajectory $\tau^*$ under the **current** dynamics model, noting that this is better than plain max. entropy RL, as it takes the task into account.
This is captured by the *Expected Information Gain* (EIG) objective, which has been proposed for the RL setting in prior work [1, 2].
In this paper, the objective is extended to account for the acquisition of data in sequence, i.e. in the form of rollouts in the real system.
To me it appears that this is a step forward from the cited prior work [1], where the system could be queried arbitrarily (at any state of the agent, coined the TQRL setting), which is unrealistic on real hardware.
One implication of this treatment is that mutual information is evaluated between the optimal future trajectory $\tau^*$ and joint sets of future states corresponding to candidate future control sequences, i.e. it serves as the objective for evaluating MPC rollouts and accounts for possible information overlaps into the future.
The method is called *Trajectory Information Planning* (TIP).

**References**

[1] Mehta, V., Paria, B., Schneider, J., Ermon, S. and Neiswanger, W., 2021. An Experimental Design Perspective on Model-Based Reinforcement Learning. arXiv preprint arXiv:2112.05244.

[2] Neiswanger, W., Wang, K.A. and Ermon, S., 2021, July. Bayesian algorithm execution: Estimating computable properties of black-box functions using mutual information. In International Conference on Machine Learning (pp. 8005-8015). PMLR.
strengths_and_weaknesses: **Strengths**
- The posed exploration objective appears sound and widely applicable. I also find it fitting, as it is about reducing the predictive entropy of the optimal trajectories MPC will actually produce, by virtue of training the dynamics model on data that is informative about them.
- While the core idea of the objective is not novel (see [1]), I believe the extension to realistic data acquisition, i.e. in sequences, is a step forward.
- The authors make it abundantly clear that information overlap in the acquired data sequences needs to be accounted for, and the method complies with that.
- Motivation is easy to follow across the manuscript, exposition is clear.

**Weaknesses**
- The method hinges on the assumption that the posterior over the dynamics model can be updated in closed form (eq. 6 and 7) and that the entropy of predictions is easy to evaluate. This is required for the evaluation of the posed MI objective. Therefore the method limits itself to GP dynamics (just like in [1]), black-box neural dynamics models would encumber the evaluation of the objective.
- GPs can be limiting both in terms of computation and expressivity for high-dimensional state spaces;
- The authors use a variation of the cross-entropy method (CEM) for MPC optimization. MC-based optimization techniques are easy to apply, but might scale poorly to high-dimensional control spaces.
- While I find the selection of experimental tasks reasonable, a lot of the environments are low-dimensional and with nice and smooth dynamics. How would the method fair on something more complex, like the ant locomotion task? Would the GP still be able to capture the transition accurately? For me it remains an open question whether the scheme is applicable to more complex systems (i.e. scalability).
- The mutual information objective is formulated for the optimal trajectory $\tau^*$ w.r.t. the current dynamics posterior. I think it is important to highlight that the "optimal trajectory" in that sense is not the ideal trajectory, but the best one available based on the current model fit. So overall acquisition success is still subject to the reachability properties of the state-space of the system.

Overall I found this is a good contribution that explores an interesting idea already introduced in prior work [1] further. Thus I lean towards acceptance.

**Minor remarks**
- The qualifier "simply" appears too often in the text (11 times).

questions: - The **MPC** ablation variant from section 5, i.e. a GP + MPC with no exploration considerations, seems to perform very well in terms of data efficiency. This is expected, as the interpolation properties of GPs are a strong prior and aid generalization when the system dynamics behave nicely in the Lipschitz sense (PILCO is a good example of this). Do you believe the TIP objective would work just as well if this aspect was not there, i.e. without the inherently fast adaptation of a GP due to its interpolation properties?
- Can you further elaborate on why TIP performs better than BARL for the plasma control tasks? The text mentions that this could be due to the inefficiency of sampling a finite candidate set for BARL, do you believe that ends up being worse than the inherent proximity of sequential trajectory states acquired by TIP?
- I don't think I agree with the statement that the median amount of data used by algorithms to solve the task is equivalent to an MPC controller with ground-truth dynamics, as stated in sec. 5 -- what is the justification?
- Have you considered using a critic, to account for infinite horizons?

Also see my remarks above.

limitations: I don't see any major negative societal impact. The authors acknowledge the main methodological limitations of the method in the conclusion, although these could be emphasized further.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors develop Bayesian Model-Predictive Control in which a new cost-function is proposed. The idea of the cost-function goes back to the information gain notion. The aim of the paper is to make the algorithm more sample efficient. 
strengths_and_weaknesses: The problem considered in the article seems relevant and the algorithms proposed by the authers seems interesting and original. The experiments show that, apparently, the algorithms are the most effective in terms of using samples. Nevertheless, in my opinion, the Weakness of the article is its descriptive part. To describe the ideas, the authors use mathematical language very carelessly, which can cause the reader serious problems in understanding the article (for more information, see the questions section).  Thus, the article seems to be promising, but requires significant refinement, in particular, in the mathematical presentation – it should be more rigorous.
questions: 1. There is no description what is $P(T|D)$ in formula (2). Also it is not clear how distributions differ for open-loop and closed-loop problems.

2. In the end of Algorithm 1, it is unclear how to create policy $\pi_g$ for posteriory distribution $P (T' | D)$.

3. On lines 203 and 204, the use of the notation $S$ for pairs, and $S'$ for one state does not seem quite intuitive to understand. Maybe it's better to use another letter instead of $S$?

4. In formula (5), it is not clear what is the result of $D \cup S'$  and what symbol $p(S'|S,D)$ stands for. When $D$ – is the set of triples, $S$ – is the set of pairs, $S'$ - is the set of single states. What is the connection between $D$ and $S$?

5. In formula (6), it is unclear what is the result of union $D \cup \tau^*$, when $D$ – is the set of triples, $\tau^*$ is the set of pairs of states and actions and a final state.

6. Perhaps, since line 207 gives an idea of a new cost function, it would be better to give a simple example when it works better than the previous ones.

7. In line 210 the non-obvious equality $C_{τ^*}(S) = −I(S', τ^*) = −I(τ^*, S')$ is given. It would be better to prove it or provide the link to the proof. 

8. In formula (7) it is unclear what is the result of union $D \cup \tau^∗_{ij}$, when $D$ – is the set of triples, $\tau^∗_{ij}$ - is the set of pairs of states and actions and a final state.

9. In algorithm 2, it is unclear how to compute the joint posterior covariance matrices. If a gaussian process is used for this, then it would be better to specify what is the kernel function.

10. What hyperparameters were used to run experiments and baseline algorithms (so that they could be repeated)?
limitations: The article does not have potentially negative societal impact.
review_title: null

Meta-review: metareview: All reviewers acknowledged to have read the rebuttal. Reviewer iWun's reply isn't visible to the authors (posted too late), see end of metareview. The most important concerns of the reviewers have been addressed by extensive replies and additional experiments. Overall the method is sound and performs well. As acknowledged by the authors, the method comes with inherent limitations to low-to-medium dimensional problems through the use of GPs. The method is useful on its own, and serves as proof-of-concept for the overall idea also for different function approximators -  but replacing GPs will require quite a bit of additional work.

***
18 Aug 2022, NeurIPS 2022 Conference Paper2167 Reviewer iWun
"Thanks to the authors for the response.

I still don't like non-strict mathematical language of the paper, but this doesn't seem to be a problem for other reviewers. In addition, I like the results of the paper, and therefore I increase my score."

confidence: Certain
Decision: Accept