Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new model-based algorithm, called SMRL, for finite horizon episodic MDPs where the transition model is specified by exponential family distribution. Essentially, the work builds on Exp-UCRL [1] and proposes to use score matching instead of MLE to estimate the parameters of the transition model, which helps it run efficiently by eliminating the need to estimate the computationally expensive log partition function required in MLE. The proposed algorithm matches the regret bound offered by Exp-UCRL while being computationally efficient. The paper also presents theoretical proof for their efficient algorithm.
strengths_and_weaknesses: Strengths:
- Strong theoretical justification.
- The language of the paper is fine apart from some typos and uncommon abbreviations.

Weaknesses:
- The assumption that transition model belongs to exponential family may be limiting for many real-world problems.
- Although it is a theory paper, I would have liked to see how does the algorithm performs empirically vs Exp-UCRL.

questions: Q1: How can we handle the problems for which the transition model does not belong to the exponential family distribution? 
Q2: Can we approximate such transition models which do not belong to exponential family?
limitations: The main limitations is that the algorithm works on specific problems which satisfies the assumptions on transition and reward models, which limits the application of the algorithm.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies model-based reinforcement learning for episodic MDP whose transition model is parametrized by exponential families with features of state and action. 
To estimate the model parameter, the author uses the score matching technique to minimize the expected squared distance between the score functions. 
To promote exploration, the author utilizes the optimistic planning.
The author states that under some regularity assumptions, the suggested algorithm achieves $\tilde{\mathcal{O}}(d \sqrt{H^3 T})$. 
strengths_and_weaknesses: I think their research is relevant to RL community since they focus on how to design a provably and efficient algorithm for the nonlinear environment.
Beyond the existing work based on the linearity assumption for transition models or MDPs, as an extension of Exp-UCRL [10], which dealt with the problem when the transition models are parametrized by exponential families, the author proposes a more efficient method for estimating model parameters. 
Based on the score matching technique, it can be more efficient than the previous method because it does not need to estimate the log-partition $Z_{sa}$.
Also, I think the analysis of the proposed algorithm is sound and the writing is clear.

However, in my opinion, the most important part of score matching relies on how to formulate the Fisher divergence (eq 3) into an empirical score matching loss (eq SM-L). 
This result is presented in Theorem 1 and since this result is from [4], I carefully consider whether the result of this paper has significant novelty

Also, in Theorem 2, it presents the result of a self-normalized concentration guarantee when the parameter to be estimated $\hat{W}$ is vectorized.
I think this kind of result can be obtained if the problem setting $\exp(\langle \psi, W \phi \rangle)$ is replaced with $\exp(\langle vec(W), vec(\psi \phi^\top) \rangle)$.
If so, I think the bilinear structure disappears. Is there a way to express this concentration result without vectorization?

questions: (1) Before introducing Assumption 2, $\Phi(s,a), C(s'), \xi(s')$ are defined on line 184. If the author could explain these functions in more detail, it would be helpful to understand the Assumption 2.

(2) The current model parameter $\hat{W}_k$ is updated after the episode is over. However, since the current problem is a model-based setting, I think it is possible to update the model parameter on every horizon because the agent receives transition feedback on every horizon. If the agent can construct a confidence set for the true model parameter every horizon without considering the computation issue for planning, I think the regret bound might be tighter. I wonder what the author thinks about this.

minor typos
1. Line 566: I think it would be better to unify the numbering of Assumption 2 and the numbering in the appendix.
2. Line 579: $\hat{V}_n + \lambda I \succeq \lambda I$
3. Line 615: I think "Under Definition 1 and 2" should be fixed to "Under Definition 1 and Assumption 2"
limitations: I think there are no issues related to social impact. However, although this paper is highly related to theoretical part, considering that many recently published theoretical papers about model-based RL also present numerical experiments, I think it would be better if there is an experimental result in this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel method for estimating the transition model of an MDP parametrized by exponential functions, in the episodic finite-horizon model-based RL setting with bounded and known rewards. The method uses score matching that reduces to ridge regression of the known parameters of the exponential parametrization, thus eliminating the difficulties associated with estimating the partition function in MLE-based methods. An estimate of the online regret for the new algorithm is derived, too. 
strengths_and_weaknesses: The main contribution of the paper is probably the idea to apply score matching to the exponential parametrization proposed earlier, resulting in a more efficient estimation algorithm. This is a very non-obvious and original advance, and given that the investigated parametrization subsumes a very large class of systems encountered in practice, the practical significance of this advance is likely to be high. However, the paper is entirely theoretical, and it is difficult to understand the computational advantages of the proposed method without at least some kind of empirical evaluation. 
questions: What are the actually achievable advantages of the proposed method in comparison with a reasonable baseline, for example MLE? Section 5 provides a comparison in general terms, but there is no empirical verification. Maybe such a verification on prototypical test MDPs would be useful in illustrating the analysis?
limitations: The authors have addressed limitations adequately, for example they have conscientiously pointed out that their analysis does not cover LQR problems, due to their unbounded costs. I do not see any potential negative societal impacts of this work.

Minor typos:
P.4 L.136,138: "gaussian" -> "Gaussian"
review_title: null

Meta-review: metareview: This is a clear and carefully written paper with a solid mathematical contribution.  The reviewers are unanimous in supporting acceptance.
confidence: Certain
Decision: Accept