Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on model-based RL, particular in the life-long setting. It sets up the notation of minimal value-equivalent models, which are minimal-sized models which can capture all task relevant features of the environment. The authors show theoretical results that under a particular definition of partial minimal value-equivalent models, the agent does not incur a performance cost of planning using the model, but experiences improvement in computational and sample complexity. This is verified in experiments based on the MiniGrid domain. The second part of the experimental section is concerned with learning such models from interaction with deep learning architectures. The authors experimentally find that:

1. Models which use representations trained through the loss of the value function perform and scale better than models trained through dynamics prediction as well. 
2. Models trained on a variety of environments perform better than models trained on a single environment in terms of robust transfer. 
3.. Models based using representations based on value-function training are more robust to model compounding errors.
strength_and_weaknesses: Strengths:

The paper carries out thorough theoretical analysis of the performance gap of MVE models and their performance in terms of planning and complexity costs.

Weaknesses:

1. It is unsurprising that models which only encode task-relevant features would perform better than generic models, which might have multiple distractions. The difficulty is learning these models using salable architectures. The results presented here are incomplete and somewhat inconsistent with current model-based RL literature, i.e. value and reward signals are often insufficient to learn meaningful representations in more complex environments. 
2. Related to the previous point, the experiments are limited to GridWorld domains, which are relatively straightforward not realistic. It is unclear whether the experimental results presented here would hold in larger more realistic domains (probably not, based on previous works, i.e. Dreamer/SLAC).

I am willing to increase my score if more involved experiments with realistic environments/control problems and current baselines are added.


clarity,_quality,_novelty_and_reproducibility: The paper is well written with clear organization. The math is clear and easy to follow. The Appendix contains enough details to reproduce results. 
summary_of_the_review: The paper provides good theoretical analysis of model-based learning in life-long/multi-task setting, however these seem of limited applicability in more complex and realistic domains.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work introduces minimal value-equivalent partial models, which are models of the environment built on top of a minimal subset of features from the observational space and holds the property of value-equivalence property. This property ensures that the value function associated with an optimal policy obtained as a result of planning with the model is equivalent to the optimal value function in the true environment. The work theoretically analyzes the value and planning losses for such models and provides estimations for computational and sample complexity. Finally, it illustrates such insights in controlled experiments and suggests inductive biases to implement these models in the Deep RL context, with proper ablation studies.
strength_and_weaknesses: Strengths:

- The work analyzes the problem of learning dynamics models with relevant information for planning, showing an important direction to develop principled methods. Hence, it is well-motivated.

- It formalizes the class of minimal value-equivalent partial models and provides its theoretical grounding. The definitions are clear, and the assumptions made are also clearly stated.

- The theoretical understanding provides important insights in terms of value/planning losses, as well as the considerations for the sample and computational complexity. Furthermore, the experimental methodology and empirical results support the proposed theory, raising and answering the appropriate questions with the right choice of baselines and ablations.

- Lastly, the work suggests inductive biases to scale its finding for the Deep RL scenario, and conduct ablation studies to support their recommendations.

Concerns:

- There is a limitation of the minimal VE partial models worth discussing in the paper. By definition, the relevant features are directly related to the reward and the task. Therefore, these models do not seem to be well-suited for scenarios where generalization across tasks is crucial (e.g., multi-task/meta-learning), which are important in the context of lifelong RL.

- The distribution shifts analyzed in question 6 only relate to the irrelevant aspects of the environment. It would be interesting to discuss or provide empirical results for distribution shifts in a relevant feature, or even in scenarios where the set of relevant features changes over time, especially given the motivation of lifelong RL. The hypothesis is that the suggested inductive biases are not robust enough to deal with such shifts when it needs to learn the representation.

- The current related work section does not place the work very well in the literature. For instance, I believe the work should contrast with recent literature on RL in the presence of exogenous distractors [1, 2] (once it approaches a very similar problem) and planning in learned latent spaces [3].


Further Suggestions/Minor Concerns:

- It would be interesting to provide a small high-level description of how the Straight-Forward Decision-Time Planning algorithm works, besides the pseudocode. This should help in the presentation of this baseline.

- In Question 3: “D minimal…” -> “Do minimal…”

- Last line of Section 5: “compunding” -> “compounding”


clarity,_quality,_novelty_and_reproducibility: Clarity/Quality: The work clearly states the problem, its motivation, and the proposed method. The experimental part also clearly illustrates the theoretical findings. The work is well-written and didactic.

Novelty: To the best of my knowledge, this work provides a novel contribution to the theoretical understanding of a common intuition. A “common intuition” refers to the fact that some previous empirical works already provide results in the same direction for planning/exploring in learned compact latent spaces [3] or even learning from imagined trajectories [4].

Reproducibility: On the theoretical side, all theorems are accompanied by detailed mathematical proofs and all assumptions needed. In the experimental section, all results provide confidence intervals for statistical significance. There is no mention of code being open-sourced to encourage reproducibility, nor any description of the computational costs/hardware associated with the experimental section.

summary_of_the_review: The proposed work provides didactic definitions and theoretical grounding from planning in “partial” models. The experimental part also provided a good illustration of the insights presented, although more experiments are required to claim that the proposed inductive bias works for more complex scenarios. There are some concerns about questions and limitations to be discussed in the paper.

References 

[1] Efroni et. al. Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information. COLT, 2022.

[2] Efroni et. al. Provable RL with Exogenous Distractors via Multistep Inverse Dynamics. ICLR, 2022.

[3] Ekar et. al. Planning to Explore via Self-Supervised World Models. ICML, 2020.

[4] Hafner et. al. Learning Latent Dynamics for Planning from Pixels. ICML, 2019.


====================POST-REBUTTAL============================

I appreciate the authors’ efforts to clear my concerns. After considering the rebuttal and the discussion with other reviewers, I decided to change my score to {5: marginally below the acceptance threshold} for the following reason: despite having interesting theoretical contributions and experiments that illustrate them in a controlled environment, it is hard to evaluate if the proposed inductive bias (illustrated in Appendix C.5) hold for more complex scenarios. My second concern point remains open, as the paper does not provide evidence to support that this proposed inductive bias is robust to deal with more complex distribution shifts, a critical aspect of lifelong RL. This is essential to clear in the experiments once prior work often suggests that more signals are needed for effective representation learning in complex scenarios [3, 4]. As pointed out by other reviewers, this is somehow conflicting with prior literature, which makes it necessary to address for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a framework to define partial models, value-equivalence and a simple method to learn partial models which are minimal and value-equivalent to the underlying MDP. The paper presents empirical results on two grid-world environments.
strength_and_weaknesses: Strengths:

The paper is trying to address a very important research problem and the presented approach is simple to implement which I appreciate a lot. The authors have also presented theoretical results on differences in value-equivalence when learning approximate VE models.

Improvements/Clarifications:

1. What is \delta in Theorem 3? How is it related to \epsilon here?
2. In fig 3, why is 3a only over a single run? Also there are no error bars/variances in 3b and 3c.
3. In section 5.1, the text only talks about models m1-m6. But fig 3 also has m7. What is m7? For fig 3, please show results for all the 7 models in 3a, 3b and 3c? Why only sample 4 out of 7?
4. The results are only for discrete grid-world environments, which makes it unclear if the proposed approach will generalize to more realistic settings.
5. While answering question 7 (on page 9), what does the search budget refer to? Planner lookahead steps or planner time cutoff or something else?
6. The word lifelong is very misleading because it seems to indicate a continual learning flavor, which is not the case. Please consider removing the word from the title and the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. The proposed model seems novel to the best of my knowledge. I have not verified the theorem proofs in the supplementary material, but the empirical results seem to be reproducible.
summary_of_the_review: I can currently only advocate a weak accept given that the experimental evaluation is not strong enough to warrant a better score. I highly recommend removing the word "lifelong" from the paper title, abstract etc. since it is very misleading and feels like the paper is about continual learning.

Edits post-discussion phase:
---------------------------------------
I thank the authors for their clarifications. After more discussion with other reviewers and the AC, I have decided to lower my score to 3: reject, not good enough. I initially kept my original score of 6 following the authors' responses, since I do see value in the work when it comes to learning good partial models of the world. However, the authors have not addressed my concern that the work is not useful for lifelong learning. It currently only deals with a very specific kind of distribution shift which does not require acquiring any new features over time, and hence the model would not truly scale to a lifelong setting. I believe that more thorough experimentation with complex distribution shifts is required to prove that the model would indeed work in the lifelong setting before the work can be accepted.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper examines the advantages of using minimal value equivalent feature sets for state representation compared to feature spaces including irrelevant features. Value-equivalence is given if a reduced feature set leads to the same equivalent policies. The paper performs some experiments comparing partial models to complete models on basic environments with limited state and action spaces.

strength_and_weaknesses: Strong points:
* The paper defines all concepts step by step and the results are clear

Weak points:
* The paper's contribution is limited as it formalizes a relatively well-known effect.
* The paper argues much with MDPs where function approximation might not even be necessary.


clarity,_quality,_novelty_and_reproducibility: The writing of the paper is clear, and the experiments seem reproducible. 
The novelty of the paper is limited as the positive effect of removing irrelevant features is well-known and studied in general machine learning. However, the paper does neither provide a new approach to a solution nor provides a very deep theoretical analysis.

summary_of_the_review: All in all, the contribution of the paper is limited as it basically examines the effects of optimal subsets which are obvious if they can be determined. However, the paper does not really help to answer questions on how to find representation spaces which are value-equivalent. In addition, there is a strong link to causal analysis in MDPs. Causality analysis basically yields a framework for analysing whether features yield a dependency in estimating a distribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a model-based reinforcement learning (RL) method with a novel notion of minimal value-equivalent model, which encodes task-relevant features and improves computational and sample complexity.  The main strength of the paper is in its theoretical analysis that shows the losses in value and planning as well as the computational and sample complexity in using the minimal value-equivalent model.

While the proposed approach is validated with grid world tasks, it is not quite not convincing that if the proposed approach work effectively in domains more complex domains, particularly in the scenarios of lifelong learning, contrary to the claim of the paper.
justification_for_why_not_higher_score: The effectiveness of the propose approach beyond grid world, particularly in the lifelong learning scenarios (which is the main motivation of the work), is questionable.
Decision: Reject