Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents the idea of self-consistent reinforcement learning.  By enforcing "consistency" between value functions and approximate  (learned) models, a model-based learning algorithm can use synthetic experience generated from the approximate model.  This paper presents a few minor variations on this theme, and explores the results empirically.
main_review: In general, I liked this paper, although I wish the results were more compelling. I lean towards acceptance because I think this paper presents some good ideas, and that those ideas are worth exploring more.  The central idea that imagined experience can somehow be used to improve the model that it is generated from seems like a Really Good Idea, and I would like to see more work on it.

* Originality

As far as I can tell, this work is original.  While the RL community has explored many of these themes, this particular combination of ideas seems novel.

* Quality

The paper is of very high quality, blending novel technical insights, a solid positioning in the academic literature, and thoughtful experiments.  I appreciated the candid balance of positive and negative results - that's good scholarship.

* Clarity

The paper is generally well-written, although much of the more technical parts are difficult to understand how exactly all of the pieces fit together.

* Significance

The significance is mixed. While the central theme of the paper is potentially significant, and the technical development is solid, the experimental results are weak.  Often, the proposed methods perform only slightly better than competing methods.  

limitations_and_societal_impact: I do not foresee any significant negative societal outcomes, beyond those inherent to all RL research.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes to improve model-based RL methods by enforcing the model and value learned to be “self-consistent”, i.e. to satisfy the Bellman eqn where both the Bellman operator and value function are defined in terms of the model. The motivation behind this is that in classic MBRL, the model and value function are learned separately, and the value function is typically learned to be consistent with the model but not vice versa. The authors propose to enforce self-consistency by adding a "self-consistency loss” (L_SC) to the base loss and jointly learning the model and value functions. L_SC is the squared TD error with a stop-gradient on the target and is calculated using rollouts from the model. Although in MuZero/Muesli the model and value functions are learned jointly as well, the targets of the loss are all defined with respect to data from the true environment and there is not term enforcing the “self-consistency” between model and value functions.
Experimental evaluations on a selection of Atari games show that jointly updating the model and value functions with the additional L_SC term leads to improved sample efficiency as compared to Dyna and a model-free baseline. The authors also empirically study the search control problem.
main_review: ### Strengths: 
The paper is very well-written and clear. Figure 1 is a nice schematic comparison of the closely related methods and helps put the proposed approach in context. The motivation makes sense based on previous work and the proposed approach is straightforward. The empirical evaluations are thorough in looking at the different variations and their effects.

### Weaknesses/Clarifications:
The related works section (particularly the first paragraph (Line 280-292)) could have more details and comparisons. 

Question: Since the model is unrolled for many steps in the computation of L_SC and each step of the unrolling is not “grounded” to the true data, it seems that the errors in the model and the value function would interact and compound with more unrolling steps and gradient updates. Did you observe a challenge like this in your evaluations?
limitations_and_societal_impact: The authors do a good job of addressing the limitations of the method through empirical evaluations (e.g. the susceptibility of the value error to poor initialization of reward).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers the problem of model-based reinforcement learning. The authors introduce a new loss, which drives the learned model and value function to be consistent with each other, rather than the model being consistent with the real environment and the value function being consistent with the real environment and the learned model. Experiments show that the proposed self-consistency loss can improve the sample efficiency and final performance. 
main_review: The writing can be improved. The writing for method and intuition are reasonable, but for experiments, a lot of missing details make understanding it difficult. 

It might be a good idea to introduce Muesli to readers who are not familiar with it. It's hard to understand what "using only the MPO component of the policy update" means without any proper introduction to Muesli. The paper mainly uses Muesli as its baseline while the experiments also use "a variant of the Muesli agent". I didn't find a detailed description of the variant. The lack of code, even implementable pseudocode, and implementation details, together with a heavy computational resources requirement, make reproducing this paper very difficult. 

This paper only applies self-consistency to Muesli (and a grid world) in many domains with discrete control (Sobokan, Atari games, 9x9 self-play Go), while it seems to me that the self-consistent model/values are also applicable to other model-based RL algorithms in continuous control domain, e.g., MBPO. 

Eq.7: How is G computed? Please give an explicit form rather than simply citing a paper if it's not elementary.  

Sec 4.1: What's the model-free baseline? How's the value normalized? 

> (L.174) In contrast, the “direct” self-consistency update was able to accelerate learning, even compared to Dyna, for both MLE and value-equivalent models.
I didn't see a significant advantage of SC-direct over Dyna with VE models. 

Sec 4.1: Please provide a short description of the environments used (especially Sokoban). 

Fig.6 (a): 6 curves in the legend, 4 curves in plots? 
Fig.6 (d): The figure caption should match the titles of each subplot. 

The improvement of Muesli + SC over Muesli seems to be minor. 

Sec 4.2: Is the policy pi deterministic or stochastic? 

> (L.257-259) We found that self-consistency updates were helpful, even though the model was neither used for value estimation nor policy improvement...
I don't understand. In both VE aux and SC aux, the model is used to improve the value function. What does "the model was neither used for value estimation" mean? 

L.273: "by randomly choosing a fraction of the states in each batch to serve as starting points". What's the "batch" here? 

limitations_and_societal_impact: I didn't find where the authors discuss the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper offers a new approach to model-based reinforcement learning with their idea of self-consistency of models and values. In the traditional (Dyna) model-based RL paradigm, an approximate model is learned with ground truth transitions and values (and policies) are updated to be consistent with the learned model. The proposed self-consistency update however, formulates a new update to the approximate model so as to make it consistent with the current value estimates using completely imagined (virtual) experience. This update is inspired by minimizing the bellman residual of the approximate model and value for a given policy (since for the true value and model, the bellman equation is satisfied and the residual is zero). An empirical analysis is presented for a set of variations of the proposed self-consistency objective with two types of (grounded) model updates -- MLE (maximum-likelihood) and value equivalent updates. In both tabular and function approximate settings, improved sample efficiency is demonstrated by the proposed method.
main_review: Overall the paper presents a novel self-consistency objective with a thorough empirical analysis of its variants on raw performance / sample efficiency (in tabular and deep RL settings), choices of search control (i.e. policy and states to use in imagined experience) and its benefits to representation learning. While not all experiments show a clear cut benefit of using self-consistency in all the cases visited in this paper, the unbiased analysis itself is an important contribution.
As a result, I feel that this paper in its current form is a significant and relevant contribution for the model-based RL community.
 
Below are more specific strengths and weaknesses.
 
## Strengths
- The paper proposes a self-consistency update which offers a new take on model updates in model-based RL. This update is important and distinct from most prior work in that it uses imagined (virtual) experience as opposed to ground truth (real) experience, allowing for freedom to update the model without collecting additional data from the true environment. This also leads to questions about what states and policy to choose for such updates and these questions were answered in the search control strategies section (Section 4.2).
- The self-consistency update is also motivated as a regularizer and shown to promote information “flow” in the presence of noisy inputs (Section 4.3 - Information Flow).
- Three different variations of the self-consistency updates are studied -- sc-residual, sc-direct and sc-reverse with a clear explanation of how each affects updates for the policy, reward or transition model. While sc-direct is the primary method, the ablations are also evaluated in the experiments providing valuable insights for future work in this area.
- The benefits of self-consistency to representation learning are insightful, especially given that the self-consistency is used as a purely auxiliary objective where the model was not used for value estimation or policy improvement.
- The evaluation of the method on Atari and Go show a clear sample efficiency improvement of the objective on top of the baseline methods.
 
## Weaknesses
- The motivation for using value equivalent model learning in addition to MLE for the model-update choice could be made more clear in the paper. Currently, I was able to understand the connection to value equivalence based on the fact that both self-consistency and value equivalence take inspiration from the bellman operator or bellman equation for their updates. However, the following statements were difficult to understand.
    - (a) Lines 76-78 (“Value equivalent models could also … in literature”) -- I have no idea how this can be achieved without further details. This sentence seems to support the inclusion of value equivalence as a major part of this paper so further details would help. 
    - (b) Lines 124-126 (“Further, if grounded … value-equivalent setting”). It is not made clear how the policy dependence of the transition model can allow for synergy with the proposed self-consistency update. Is it because of the information flow? Further details on this would help as this is another important justification for the use of value equivalence.
    - Overall, the link between value equivalence and self-consistency is not explicitly addressed but rather is inferred after reading the above two sentences. It is also missing from the introduction. I would recommend addressing the link and the motivation for using value equivalence early on in the paper before existing value equivalent methods are introduced.
 
- The experiment on information flow in Section 4.3 is an important contribution despite the lack of statistically significant differences between Dyna and SC-Direct. However, I do not understand the justification for the poor reward estimates affecting self-consistency more while the poor value estimates affect Dyna more. The inversion of trend from reward noise to value noise is the issue here. The trends mostly have overlapping CIs other than the first figure, so I would recommend either dropping any strong claims on difference in performance or making the justifications for these trends more clear.
 
- Line 312 in conclusion (last sentence): This future work sentence is not clear. Why is there a need to perform policy improvement for value equivalent model-based agents in imagined experience? Don’t existing value equivalent model-based agents already perform policy improvement in this way e.g. [11]?

limitations_and_societal_impact: The authors have sufficiently addressed limitations of their work. For example, Section 4.3 demonstrates the effect of noise in reward or value estimates on their method in comparison to the Dyna baseline.
review_title: null

Meta-review: The paper proposes a self-consistency approach to model-based RL. The self-consistency refers to jointly optimizing the value function and the model such that (a variant) of the Bellman residual is minimized.

The reviewers are all positive about this work. They believe that it is an original work and written (mostly) clearly.  Most of their concerns are also addressed during the discussion phase, and there is no concern that prevents them from accepting this work. Therefore, I would recommend the *acceptance* of this paper.

There are some minor issues that can be improved though. Please refer to the reviews for more detail. I enlist a few of them:

- Improving the writing, especially in the Experiments section, in which some details are missing.
- Explaining and studying the effect of the number of unrolled steps in the computation of L_SC.
- Some of the reviewers believe that the paper isn't very reproducible. Please try to improve it in this regard, by providing more detail or releasing (some parts of the) source code.
- Better explanation of the relation between value equivalence and self-consistency.
Decision: Accept (Poster)