Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Based on the concept of operator splitting, the authors analyzes convergence properties of value iteration with approximate dynamics models.
strengths_and_weaknesses: The theory looks sound.

In addition to the analysis, a new algorithm, operator splitting Dyna is proposed.
questions: Does the theory apply to continuous state-action cases?
limitations: Empirical evaluation is only with a simple cliff wall task.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors consider a novel Value Iteration by Operator Splitting for both Policy Evaluation and Control problems. Further, they show a faster convergence rate when the learning model is accurate compared with general RL methods with fixed $\gamma$. Then, they give the error bound of the proposed OS-VI and develop OS-Dyna. The experimental environment is simple but effective to show the performance of OS-Dyna.
strengths_and_weaknesses: Strengths
	A novel value iteration formulation to balance Dyna and MFRL.
	Precise problem formulation and derivation structure;
	Reasonable experiments and demonstratable results;
	Clear structure and understandable writing.

Weaknesses
	As mentioned in 2.2, the dynamic error $e_k$ are (norm-) convergent when $\|G\|=\|M^{-1}N\|<1$. When the OS-VI method is proposed, the authors should also discuss the convergence for $M^\pi$ and $N^\pi$ in 3.1, even though they can only be discussed in the linear systems.
	In Theorem 1, the authors give the effective discount as $\gamma’$. I wonder that is there such a situation about $\gamma’>1$. And, if $\gamma’>1$, the proposed Varga operator may not be satisfied (3.2).
	In the proof of Lemma 4, the authors derive $S^\pi V_1- S^\pi V_2$ with some doubts. In different iteration epochs, the Varga operator may correspond to the different learning dynamic model $\hat{P}^\pi$, i.e., 
$S^\pi V_i = (I-\gamma \hat{P}_i^\pi)(r^\pi+\gamma(P^\pi-\hat{P}_i^\pi))$, which may result different result.
	In the experiments, the authors may consider putting the smooth $\lambda$ cases in the paper other than in the supplementary materials.

questions: Please see the questions in Weaknesses and the following.
	For the proposed algorithm OS-Dyna, the authors claim that the operator splitting approach is a hybrid of MBRL and model-free RL. I agree with this claim. But, I think the authors should add some analysis of the proposed algorithm from the following perspectives.
       a. Compared with the original Dyna framework, they utilize the learning dynamic model $\hat{P}^\pi$ to rollout virtual samplings for improving policy efficiency. How about OS-Dyna’s performance? 
       b. Compared with general model-free RL methods, like Soft-Actor-Critic, OS-Dyna utilize the learning dynamic model $\hat{P}^\pi$ as a reward penalty term defined in (3.3). Thus, compared with SAC's entropy policy, how does the proposed penalty term work?
       c. Can OS-Dyna be extended in non-linear systems? Like some general technologies in RL, the authors may consider the sample-based model $\hat{P}^\pi$ and replay buffer to re-sample some experiences for approximately computing the term $P-\hat{P}^\pi$.

limitations: I think that this work would be more convincing if the authors can extend the performance of OS-VI in some complex environments due to its potential performance improvement.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the problem of planning in a discounted MDP when an approximate model of the dynamics is available. The author propose an algorithm that can take advantage of such model to quickly evaluate a policy or compute an optimal one. 

Each iteration of the value iteration algorithm can consists of solving a linear system (in discrete MDPs). In the case where two models are available, the matrix to inverse can be expressed as a difference of two matrices: one involving the approximate transition, one involving the model error. It is shown that solving the original problem is equivalent to applying value iteration in a modified MDP with the approximated dynamics, and a new reward function including the model error. 

A theoretical analysis of the convergence of the proposed method is provided along with some extension to a model-based reinforcement learning algorithm. 

Experimental results on toy problems are provided and show that the proposed method allow for faster convergence in both planning and RL settings. 

strengths_and_weaknesses: Overall the paper is clearly written and easy to follow. 

The idea of considering value iteration through the lens of operator splitting is very interesting and original. The derivation is mathematically correct and very well explained. 
The main contribution is theoretical. 

Having approximate model of the environment might be common in many applications. The author also show that when no model is given, but the model is learned (OS-dyna) the method still outperform existing model based RL. 

The extension to model based RL and the learning of the new reward signal is very clear and a very interesting idea. 

It is a bit puzzling that the convergence rate can be made faster, I wish the author added an illustrative example to get a better grasp of this in addition to theorem 2. 

minor weaknesses:
- the lack of discussion about continuous state spaces. 
- The author could provide a more detailed pseudo code of the method in appendix as algorithm 1 relies on a lot of notation. 
- More discussion on how to update the model p_hat would have been useful. 

questions: 1. How does the state space (discrete vs continuous) affect the method? 
2. How do the experimental results look like for more extreme cases when the model is completely off?
3. Would it be possible to get a convergence rate for OS-dyna? In terms of number of sample to reach a certain model accuracy. 

limitations: As far as I understood, this method is only applicable to problem with discrete state spaces, so that the transition model can be represented as a matrix, but it is never clearly stated. The author should make mention of this limitation or clarify how continuous spaces could be handled. The operators are defined for continuous spaces (e.g. 3.3), but I don’t see how the matrix $(I – \gamma P)$ would be represented in such case. 

The authors clearly explains that the convergence rate is highly dependent on the quality of the approximate model.
review_title: null

Meta-review: metareview: All reviewers appreciated the strong theoretical contribution of the paper. The idea was evaluated to be very innovative and offers very good theoratical guarantees. The paper is well written and, while the contribution is mainly theoretical, it also offers a basic evaluation of the presented idea. I follow the reviewers with their recommendation.
confidence: Certain
Decision: Accept