Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a general theoretical and algorithmic framework for model-based RL in which the authors use data likelihood to estimate a model quality and use an optimistic posterior sampling for data collection.  The combination of Heliginer distance-based likelihood and posterior sampling brings some advantages for their algorithmic framework as well as for regret analysis. They show that in many special cases, the sample complexity of their proposed algorihtms improves previous related works. 

strengths_and_weaknesses: Weaknesses:
- No experiment is provided to demonstrate the efficiency of the proposed algorithm.
- The framework is designed for only the posterior sampling approach.

Strengths:
- The paper is well written. 
- The technique of combining the likelihood and optimistic posterior sampling is novel. The technique allows avoiding the minimax problem which was required in the model-free work of Agarwal and Zhang [2022], and avoiding to incorporate the Hellinger distance in the algorithm which is complicated to estimate as in the work of Foster et al. [2021]. 
- In some special settings, the sample complexity of their proposed general algorithm improves that of some previous works.  
questions: It would be nice if the authors address my some questions:
- What is the connection between the likelihood estimate at line 4 of the algorithm and the Hellinger distance defined in Definition 2 as the authors mentioned?
- Does the coefficient $dc(\epsilon, \alpha)$ depend on the size of the action space?
- Can the authors explain intuitively why their regret bound improves that of Sun et al. [2019] in V-type setting?
- Are your results able to compare with those using the UCB approach in model-based RL? 
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper develops and formulates a statistical framework for model-based reinforcement learning with an optimistic posterior sampling mechanism. The paper develops a theoretical framework to solve the exploration-exploitation balance in an MBRL framework by incorporating an exploit-or-learn behavior in the learning framework- typical of optimistic posterior sampling algorithms. The paper draws an interesting connection between the regret of the optimistic PSRL algorithm with Hellinger distance and thereby derives a unified posterior sampling algorithm with state-of-the-art sample complexity under several structural assumptions.
strengths_and_weaknesses: The primary strength lies in the statistical formulation of an optimistic model-based posterior sampling reinforcement algorithm and the connection of the regret with the Hellinger distance between the conditional probabilities to derive the state-of-the-art sample complexity for MBRL under structural assumptions. As done in optimistic algorithms, the algorithm assigns an optimistic bonus as done by $p_t(M) \propto p_0(M)\exp(\sum_{s=1}^{t-1}(\gamma V_M(x_s^1) + L_s^{h_s}(M))$ which means that the posterior would concentrate on model which predicts the history well or models under which the policy will lead to high value function. The novelty lies in the decoupling framework which builds the connection between the Bellman error and the Hellinger distance $D_h (P_M(\cdot | x^h, a^h), P_\star(\cdot | x^h, a^h))$, with a dependence on the true dynamics. As it can be observed that it has a dependence on the true dynamics, hence it is critical to understand the confidence set creation to measure how
well the prior distribution covers the optimal model $M^*$ which is shown in Definition 3. Although, it needs a little more explicit explanation on how $M(\epsilon)$ is non-empty for $\epsilon > 0$ and also the bound still holds for misspecified prior. Overall the paper proposes a very interesting theoretical framework leveraging a Hellinger decoupling framework to derive the state-of-the-art sample complexity for optimistic PSRL-type algorithms and is also simplistic and unique in its formulation. The paper is well-written and the mathematical analysis is clear and precise. The main strength lies in the decoupling framework which is novel and very interesting and helps in controlling the Bellman error optimally. However, as pointed out by the authors it will be interesting to see the empirical evidence of the algorithm especially given that the theory has a strong dependence on the Bellman error and recent literature has shown that the Bellman error is a poor proxy for the accuracy of the value function [1]. 

References : 
[1]. Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error by Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, Shixiang Shane Gu. https://arxiv.org/abs/2201.12417


questions: 1. How does the bound in Definition 3 holds even under misspecified prior, can you please discuss that clearly?

2. How $\eta, \eta'$ in 4 in Algorithm 1 is obtained, and how much impact will it have on the analysis?

3. In [1], they show that the Bellman error is a poor proxy for the accuracy of the value function and the magnitude of the Bellman error is weakly related to the distance to the true value function. However, here the analysis has a very strong dependence on the Bellman error. Although it's a general issue and not specific to this research, please can you share your thoughts on the same.

limitations: The authors have mentioned the limitations of the current theoretical analysis including the need to perform empirical evidence, extension to non-linear function approximations in the Conclusion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work proposed a unified framework for model-based RL by designing likelihood based posterior sampling methods. The theoretical framework unified many previously studied example scenarios and give state-of-art sample complexity in some cases. The authors also claim by combining variance reduction techniques, the sample complexity can be further improved.
strengths_and_weaknesses: The paper discusses a meaningful unified approach for model-based RL. The algorithm and results are well presented. The theoretical analysis looks solid.
questions: 1. Extra T in Eq. (3)?
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a general algorithm for model-based RL based on optimistic posterior sampling. Theoretical analysis is provided to show that it achieves state-of-the-art sample complexity under some structural assumptions. 
strengths_and_weaknesses: Strength: The proposed algorithm is an interesting effort to solve the problems in model-based RL that collects data from the environment given the current learning state and defines the quality of a model given the current acquired dataset. The idea is to use the data likelihood as the estimate for model quality, and use an optimistic posterior sampling method to collect data. A so-called Bellman error decoupling method is used for the sample complexity of near-optimal policy, and it is sample efficient under the condition that decoupling coefficient of a model-based RL is small.

Weakness: The assumptions in the sample complexity is somewhat stringent for the Hellinger decoupling. It's not clear under the general settings how to measure the Hellinger distance and Bellman Error to quantify such condition. It help better understand such condition if the authors can provide further discussions on how we can reasonably quantify (or even estimate) these quantities in order to better understand the sample complexity.
questions: It's related with the weakness discussed above. Can the authors provide further discussions on how we can reasonably quantify (or even estimate) these quantities in order to better understand the sample complexity in some general settings.
limitations: Yes
review_title: null

Meta-review: metareview: The paper proposes a new theoretical framework to design posterior sampling methods for model-based RL. All the reviewers agree that the theoretical framework is novel and can avoid many complications in the previous works. 
confidence: Certain
Decision: Accept