On Rollouts in Model-Based Reinforcement Learning

Authors: Bernd Frauenknecht, Devdutt Subhasish, Friedrich Solowjow, Sebastian Trimpe
Primary area: reinforcement learning
Keywords: Model-Based Reinforcement Learning, Model Rollouts, Uncertainty Quantification

Abstract:
Model-based reinforcement learning (MBRL) seeks to enhance data efficiency by learning a model of the environment and generating synthetic rollouts from it. However, accumulated model errors during these rollouts can distort the data distribution, negatively impacting policy learning and hindering long-term planning. Thus, the accumulation of model errors is a key bottleneck in current MBRL methods. We propose Infoprop, a model-based rollout mechanism that separates aleatoric from epistemic model uncertainty and reduces the influence of the latter on the data distribution. Further, Infoprop keeps track of accumulated model errors along a model rollout and provides termination criteria to limit data corruption. We demonstrate the capabilities of Infoprop in the Infoprop-Dyna algorithm, reporting state-of-the-art performance in Dyna-style MBRL on common MuJoCo benchmark tasks while substantially increasing rollout length and data quality.

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729037594898

thread_type:
official_review

<thread_object>
creator: reviewer_anzx

<content>
summary:
The author presents a new method called *Infoprop*, which can be used as a plugin for Dyna-style model-based RL. The plugin increases the quality of data rollouts from the estimated model, thus enhancing the Dyna-style model-based RL. The experiment in Mujoco showed the SOTA performance except for the humanoid environment. The author also provides a solid theoretical analysis of why the plugin can increase the quality of rollouts based on the estimated model.

strengths:
The author developed a new method, Infoprop, that enhances the performance of Dyna-style model-based RL:

*Versatility*: The Infoprop method can be integrated as a plugin for Dyna-style model-based RL, making it a flexible and adaptable enhancement for existing frameworks.

*Improved Data Quality*: By increasing the quality of data rollouts from the estimated model, Infoprop enhances the overall performance of Dyna-style RL approaches.

*Experimental Results*: The experiments conducted in Mujoco environments demonstrate state-of-the-art (SOTA) performance, showcasing the effectiveness of Infoprop in most environments, with the exception of the humanoid scenario.

*Theoretical Foundation*: The method is backed by robust theoretical analysis, explaining how the plugin improves rollout quality from the estimated model, which adds credibility to the approach.

weaknesses:
1. The theoretical results in Section 4 are difficult to follow. A summary of the main result should be given at the start of the section. Also, I suggest the author clearly present the main result as a theorem. 
2. Too many notations have been introduced, which could be simplified. For example, Section 2.4 could be condensed into a single sentence, accompanied by equation (5) and relevant citations since it represents a well-known formulation in model-based RL.
3. The enhancement applies specifically to Dyna-style RL, not all model-based RL approaches. Making the title misleading.
4. The Assumptions at the end of Section 4.2 are stated without any explainations.

questions:
In Figure 6, INFOPROP Dyna does not outperform MACURA. However, if I understand correctly, the proposed method can be used as a plugin within MACURA, which makes it puzzling that INFOPROP Dyna does not surpass MACURA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729914525953

thread_type:
official_review

<thread_object>
creator: reviewer_ufds

<content>
summary:
This paper proposes a model rollout method for model-based reinforcement learning called infoprop. The main idea is to control for epistemic uncertainty using model rollout to reduce model error accumulation and in turn improve value or policy optimization. The concrete method is a re-weighting of ensemble predictions and an entropy-based criteria to stop rollout. The Mujoco experiments demonstrated substantial improvement in reducing error accumulation as well as convergence speed.

strengths:
The proposed method is original and sound. The experiment hypotheses are very clear the the results clearly demonstrated the advantage of the proposed method.

weaknesses:
The paper could benefit a lot from intuitive explanations of the proposed approach. For example, what's the intuition of the covariance intersection fusion result for readers that are not familiar with this literature.

questions:
* How does eq 14 actually follow from eq 4 and 9? 
* When the authors say eliminate epistemic uncertainty, does that mean setting the last term in eq 9 to zero? If so is the rest of the derivations simply accounting for $\mu^{\Delta}$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667605974

thread_type:
official_review

<thread_object>
creator: reviewer_nha5

<content>
summary:
This paper presents a rollout mechanism that uses information theory to reduce the impact of epistemic uncertainty in MBRL. The paper presents some experimental analysis and a derivation of a result related to information loss.  The paper uses the idea of marginal entropy and mutual information to determine data corruption and stop a rollout.

strengths:
Soundness
======
The paper uses the idea of marginal entropy and mutual information to determine data corruption and stop a rollout in MBRL. The idea is presented carefully with a derivation for the main result.
  
Significance & Related work
=========
The paper has a brief related work section, as part of the main paper, which is appreciated. However, there is a need for a deeper analysis of existing work, including  more in-depth comparisons that show the benefit of Infoprop to others, specifically including work in using simulated rollouts to guarantee performance, robust MBRL (Sung 2024, Kuo 2021), risk sensitivity (e.g. Webster 2021) and other uses of information theory in MBRL.

Presentation
=========
The paper is well written with few typos, usually around empty spaces in the text.

Quality
======
My main concerns are around the quality of the experimentation (see below) and on the need for authors to explore the limitations of their approach a bit deeper before publication. For example, the authors mention that the rollouts exhibit excellent data quality and yet there are instabilities: is this a function of the integration mechanism or of overfitting?

weaknesses:
Experimentation
=========
The experimental analysis show the benefits of Infoprop when it runs in simple environments, however deeper experimentation is needed. For example, I would have expected to see some ablation studies showing the impact of the choice of lambda, instability, overfitting and other issues. 

Overall the paper presents a very promising initial idea with several avenues of experimentation that need to be explored to ensure its maximum potential is reached.

questions:
The main limitations of the work, as identified above, are in the quality and significance as discussed above. These could be alleviated with a more in-depth experimentation which would lead to a more mature solution, and a more in-depth discussion of related work. All of them require space, and a such my questions relate to the guidance for the discussion section.

Specifically:
1. where is overfitting most prominent with Infoprop?
2. where there other information theoretic metrics (e.g redundancy) considered in the Infoprop design? If no, why not? if yes, why were they discarded?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710640195

thread_type:
official_review

<thread_object>
creator: reviewer_j8se

<content>
summary:
This paper proposes Infoprop, a rollout mechanism designed to improve sample efficiency and asymptotic performance in model-based reinforcement learning (MBRL). Infoprop addresses the issue of epistemic model uncertainty typically introduced by ensemble methods or dropouts. It projects sampled next states into the maximum likelihood distribution of the predicted next state distribution and uses an information-theoretic criterion to determine the termination point of rollouts. When applied to Dyna-style MBRL, Infoprop demonstrates state-of-the-art performance, as shown in experiment results.

strengths:
1. **Significance**: Model error remains a significant challenge in MBRL, and Infoprop addresses it through the rollout process.
2. **Extensive Experimentation**: The authors provide comprehensive experiments and analysis, including detailed evaluations of algorithm trajectories and environment buffers, which validate the effectiveness of the Infoprop algorithm.

weaknesses:
1. **Assumptions**: The approach relies on two key assumptions. It is important to empirically assess the extent to which these assumptions hold in experimental environments.
2. **Organization**: The theoretical section could benefit from clearer structure. Presenting the main results in the form of definitions or theorems could improve readability. Simplifying technical details and focusing on the implementation—particularly regarding Equation (14)—would make the logical flow smoother.

questions:
1. Is the model error come solely from epistemic noise?
2. What is the advantage of using AES models, and why do you believe Infoprop does not impact their effectiveness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731951686777

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
Dear reviewers, first of all, thank you for the helpful, constructive feedback. We have uploaded a revised version of the paper. Here the first 26 pages correspond to the new paper with appendix. In case you prefer reading the diff file, this corresponds to page 27 onwards.  We hope the new version can address your concerns. In the following, we want to address concerns that were mentioned by several reviewers. Replies to individual requests can be found in the respective individual rebuttals.
We are grateful for the excellent comments, which have substantially improved the readability of our paper.

**1) Section 4 is hard to follow:**
We have substantially revised the presentation of Section 4 according to your comments, i.e. we chose a more formal presentation that provides the core results in definition, lemma, and theorem environments while moving technicalities to the appendix. In particular, the key results of the Infoprop rollout are presented in Theorem 1 and the key idea of Section 4 is given at the beginning supported by a figure that builds intuition. 


**2) Meaning/Soundness of Assumptions not discussed.**
The last paragraph of Section 4.1. (end of page 4) now discusses the assumptions and why they are valid empirically. From model rollouts e.g. in Figure 1, we see the models do not underestimate stochasticity due to aleatoric uncertainty and have negligible bias.

**3) Presentation of the Infoprop state (former Eq.14) is confusing.**
We have decided to remove unnecessary technical details from the presentation of the Infoprop state in Definition 5. Instead, we provide an additional step-by-step derivation of the Infoprop state in Appendix D3.
Thus, we aim to convey a better intuition, while still providing the technical details.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Infoprop, an original method with a rollout mechanism designed to improve sample efficiency and asymptotic performance in model-based reinforcement learning.
The reviewers appreciate the theoretical foundation of the method, the originality of the method, as well as the presented experiments.
There are some concerns about having a more thorough evaluation of the limitation of the method, particularly regarding the reported instabilities.
There are some comments regarding the writing style, although these could be improved and are not fundamental about the method.